Got it sorted out!! It was a DNS issue connecting to the Memcached servers via hostname instead of IP address. Even adding them to the hosts file didn't fix the issue. Switched over to IP and there was a ten-fold improvement.
That would mean that there is a delay somewhere in the name resolution process. Check the flow of the DNS traffic. On networks where there are several DNS "hops" to get to the authoritative zone, I've seen (on occasion) where DNS adminstrators have mistakenly used forwarding and/or conditional forwarding incorrectly and have introduced extra "hops" which result in slower queries.
Even adding them to the hosts file didn't fix the issue
That doesnt make sense. Once you add the host name to the HOSTS file, this bypasses the need to query a DNS server for resolving a query to that host name. I am not familiar with Linux, but on Windows, that action causes the hostnames to be immediately loaded into cache.
Are you sure it wasnt a fluke? You may want to try to reproduce it, and if it comes back, take a packet capture, assuming you are interested in all that extra work.
hmm... You can definately measure a difference between performing a DNS lookup vs accessing the IP directly. I wouldnt expect 500ms unless there was a problem (with too many hops, or network latency), but that's fine.
I am still wondering why you would still see any latency when you tried using the HOSTS file (not that I would recommend you maintain a HOSTS file unless its a machine or two).
In any case, it may not be worth your time to find the actual issue if its working good now. I am sure that your brain is in overdrive working on more new DaniWeb features...