CMG: Free Performance Data and White Papers

EddieC 0 543 Views

Which hypervisor performs better, Xen or VMware's ? That apparently depends on which organization you ask. But for a team that's tasked with choosing a virtualization platform, some impartial data would sure be helpful.

"That's where we come in," said Michael Salsburg, director of the Computer Measurement Group, a non-profit that acts as a repository for the performance data gathered by hundreds of member companies around the world. We spoke recently on the phone after a colleague told me about CMG.

In “,” an article coincidentally written by Salsburg, we learn that a report published by VMware in January, 2007, showing its product greatly outperforming Xen did not tell the whole story. A subsequent report released by the Xen project in April of that year (six months prior to its acquisition by Citrix) showed benchmarks on par with ESX.

This is but one example of the thousands of performance and best practices papers collected since CMG's founding in 1975 as a place for IBM mainframe users to store performance analysis data. It quickly expanded to include Linux, Unix, Windows, networking and storage. "Members of CMG have a reputation as experts at making sure that levels of service end-to-end are what the business wants,” said Salsburg. And it's all free for the taking. "We see ourselves as a go-to community where you can see what’s out there and how products are performing," he said. "And that’s what we want people to know about.”

CMG also organizes CMG '08, an international conference at which Salsburg said about 500 people learn what he called the "three R's" of performance testing. "Those are measurement, analysis and forecasting. How do you instrument, how do you measure what you get back, and how do you model and forecast what you measured? We teach you to do what-if analysis” to understand for example how the application would behave if the workload is suddenly three times greater than expected, he said.

The main idea, Salsburg said, is to give IT "something you might not be able to create in your test environment and can’t afford to duplicate your production environment." Anyone with enough money can get any performance they want, he said. "The balance is to get the best performance for the least amount of money."

Dani AI

Generated

Good point, — vendor white papers are useful background, but they rarely answer "how will my workload behave?" The practical approach is: define the exact workload you care about, run repeatable tests on identical hardware and configuration, and compare latency percentiles and resource overhead (not just peak throughput).

Do this in three stages:

  • Prepare an identical testbed for each hypervisor: same firmware/BIOS settings, same kernel/drivers, same storage/network paths, and document every tuning change you make.
  • Stabilize the environment (disable aggressive power-saving, set CPU/NUMA pinning where appropriate) and run steady-state tests long enough to avoid transient caches and warm-up effects.
  • Capture the full distribution: P50/P90/P95/P99 latencies, IOPS, throughput, CPU/sys time, interrupts, and storage queue depths. Percentiles and histograms reveal problems averages hide. (brendangregg.com)

Use flexible, reproducible tools and keep raw outputs for later analysis. For IO, fio is the de facto flexible tester (job files + JSON output make automation easy). For database/OLTP-style workloads, sysbench gives workload scripts and percentile reporting. Example fio job (random read/write 4k, 70/30 mix, time-based):

[global]
ioengine=libaio
direct=1
time_based
runtime=300
group_reporting
size=4G

[randrw]
rw=randrw
rwmixread=70
bs=4k
iodepth=32
numjobs=4

Run a comparable OLTP run with sysbench, e.g. sysbench --threads=16 --time=300 oltp_read_only run. (github.com)

When reading third-party or vendor papers, check for reproducibility: full hardware list, hypervisor and kernel versions, exact workload descriptions, tuning/firmware changes, and whether results report percentiles and statistical variance. For a vendor-neutral baseline, consider standardized suites such as SPEC virtualization benchmarks as one more datapoint — but still validate on your real workload. (spec.org)

Summary: trust reproducible tests that mirror your production load, capture distributions (not just averages), keep raw data and configs, and prefer repeatable tools so results can be independently validated.

Be a part of the DaniWeb community

We're a friendly, industry-focused community of developers, IT pros, digital marketers, and technology enthusiasts meeting, networking, learning, and sharing knowledge.

Edit Preview