There have been intermittent issues with CHM on vista. The most common:
1) You may not have a real CHM file, it may be HLP (WinHelp) file rename (HLP is not supported on VISTA). If you have another machine running XP, you could try reconverting it using converter (hh.exe).
2) The CHM viewer component may not be registered. To fix this, go to your Run/command prompt and type "regsvr32 hhctrl.ocx" (and enter). If you got "OK" - try to open file again. It may work. If you didn't get "OK", it is something else.
3) You may have security update that blocks CHM files (because it is compiled HTML, there is a chance it might contain scripts). It might be blocked by system policy for the same reason if you are in a network. To solve it right click on CHM file and select Properties. On the General Tab click Advanced, then Unblock, then Apply. If you have unblocked the file successfully, the unblock button disappear.
Whatever is on my Windows Vista works with the .chm files I have. I just want to know what the darn thing is actually called, for instance 'something.exe' rather than the nebulous 'Microsoft® HTML Help Executable'. I need the actual exe file to run the language specific chm help file from my programming editor.
Actually, a friend just told me that her Windows XP machine gives her the same dimwitted answer in the properties window. There's got to be an executable file somewhere that takes .chm files as argument.