Which hypervisor performs better, Xen or VMware's ESX? That apparently depends on which organization you ask. But for a team that's tasked with choosing a virtualization platform, some impartial data would sure be helpful.
"That's where we come in," said Michael Salsburg, director of the Computer Measurement Group, a non-profit that acts as a repository for the performance data gathered by hundreds of member companies around the world. We spoke recently on the phone after a colleague told me about CMG.
In “Xen vs. VMware – The Battle of Brands,” an article coincidentally written by Salsburg, we learn that a report published by VMware in January, 2007, showing its product greatly outperforming Xen did not tell the whole story. A subsequent report released by the Xen project in April of that year (six months prior to its acquisition by Citrix) showed benchmarks on par with ESX.
This is but one example of the thousands of performance and best practices papers collected since CMG's founding in 1975 as a place for IBM mainframe users to store performance analysis data. It quickly expanded to include Linux, Unix, Windows, networking and storage. "Members of CMG have a reputation as experts at making sure that levels of service end-to-end are what the business wants,” said Salsburg. And it's all free for the taking. "We see ourselves as a go-to community where you can see what’s out there and how products are performing," he said. "And that’s what we want people to know about.”
CMG also organizes CMG '08, an international conference at which Salsburg said about 500 people learn what he called the "three R's" of performance testing. "Those are measurement, analysis and forecasting. How do you instrument, how do you measure what you get back, and how do you model and forecast what you measured? We teach you to do what-if analysis” to understand for example how the application would behave if the workload is suddenly three times greater than expected, he said.
The main idea, Salsburg said, is to give IT "something you might not be able to create in your test environment and can’t afford to duplicate your production environment." Anyone with enough money can get any performance they want, he said. "The balance is to get the best performance for the least amount of money."