What is the difference in memory management in openvms and mac os x?

OpenVMS is a virtual machine manager that uses the host OS services for this stuff (usually). As for the differences, that is very difficult to say. Contact OpenVMS for more information - provided they want to give it to you. If it is "open", then the source should be available for your perusal and analysis...

Also, note that OSX is basically BSD Unix, which is ALSO open source. Apple adds their UI cruft and some additional software, but the memory management functionality should be pretty standard.