During peak times, the server load average goes crazy high, and PHP has intermittent problems connecting to MySQL. Right now, both PHP and MySQL are on the same server, and adding a second database server is something for the near future but not a possibility at this moment. Would adding more RAM help the issue? (Currently 2 gigs, would upgrade to 4) Or is the only thing that would help server load average be to have more cpu cycles?
I assume that you are refering to the load averages that are reported on a Linux system. If so, the load average represents the average number of processes that are requesting CPU cycles (i.e. processes that are not in a wait state or dead) over the last minute, five minutes, and fifteen minutes. If your load is much higher than you want it to be, there are a number of possibilities on how to reduce it. If the processes are bottlenecking on RAM usage and you have a high swap rate, I would say that RAM would definately help you. If you don't have a RAM bottleneck and a high swap rate, your processor is the bottleneck. Putting a faster processor on the machine will help, but it may not be the best solution.
You can gain a lot more out of your current setup and reduce the load by optimizing your setup. Optimizing your webserver setup, database queries, database indexes, and complex code segments can create tremendous speed improvements. There was a time that I reduced the load drastically on one of our machines just by adding some much-needed indexes to our database tables.
Another possible bottleneck is your disk I/O. It's possible that your disk is causing much of the extended execution times of your processes. If this is the case, creating a RAID setup can improve your disk access times but can also add headaches of their own.