You install stuff remotely using ssh and apt-get/aptitude, no gui required, just make sure you prefix the commands with sudo though, since both apps require root privileges.
TBH, I would say that if you just want to use firefox, OpenOffice and similar apps, then remote X might be enough if your connection speed isn't a problem.
This can be used like:
ssh -X user@server <command>
where command could be firefox or openoffice or anything else, the app would then appear on the X display of your client Linux machine (or windows machine if you have something like cygwin+X or Xwin32 installed)
You may find response a little slow, nomachine gets around this fairly well, since it compresses raw X, rdp or vnc connections.
If you want a full terminal server emulation, so you can connect and then see a normal login screen, you are probably going to want to use something like vncserver and XDMCP.
http://ubuntuforums.org/showthread.php?t=795036 or
http://news.softpedia.com/news/Setti...rt-52875.shtml may shed some light on it.
When I said 'attack surface' what I mean is that the more that is installed (and X and all the bells and whistles that go with it are a large install) the greater the risk that a bug in any part of it could allow a compromise. The public/private key isn't a total solution, since that only solves the problem of automated bruteforce attacks for ssh. The keys themselves maybe insecure (see the debian openssl bug recently) or there may be bugs in sshd or any other server process (anything that allows remote machines to connect to it).