If it doesn't listen on localhost, it won't work.

Secondly, use iptables to allow/deny connections on a different interface if you don't want to allow connections to a specific interface.

FYI, "interface" is NOT the same as IP address, since you can have numerous IP addresses bound to the same interface (eth0, eth1, etc.)

Yes I figured vnc4server wouldn't work unless it listened on localhost, I fogot to post back here.

I switched to x11vnc, when combined with xvfb allows me to start the server on any interface.

Yes I am aware that interfaces are not the same as ip addresses, however some servers use ip addresses to idenfity the interface they will listen on, such as Bind
And some servers use the interface to identify the interface, such as Samba,

Yup, I know the difference, but the way you originally worded it made me unsure of whether or not you understood the differences yourself.

(Incidentally, I thought the answer you got from the VNC list back on the 8th was sufficient.)