Virtualization shoot-out: Citrix, Microsoft, Red Hat, and VMware

Virtualization shoot-out: The test bed
The fine folks at Dell were kind enough to lend us a bunch of high-end gear to run all of our tests. We requested blade servers for a variety of reasons. Primarily, we wanted the ease of setup and configuration offered by the blade chassis, which consolidates the power, network, and remote management into a single unit. We chose the two-socket blades for our test servers, as these are more representative of production hypervisor configurations than other CPU densities.

We were equipped with a Dell PowerEdge M1000E chassis with two Dell PowerConnect M8024 10G switch modules and a PowerConnect M6220 gigabit switch module. The storage tasks were easily handled by a Dell EqualLogic PS6010XV 10G SAN array, and we used four Dell PowerEdge M710HD blades to run the hypervisors. Each M710HD was equipped with two Intel Westmere 5645 CPUs running six physical cores at 2.40GHz. Those were accompanied by 96GB of DDR3 RAM, dual-port Intel X-520 10G Ethernet mezzanine adapters, and built-in dual-port gigabit NICs. Each server also had Dell's redundant SD-based flash devices for embedded installations and a pair of 72GB SAS drives in a RAID1 configuration for hypervisors that required traditional installation.

For backline duties, we used two Dell PowerEdge M610 Intel Nehalem-based blades. These blades were not part of the actual test, but were used to provide supporting services such as Microsoft Active Directory, DNS, and DHCP. Suffice it to say, we were very well outfitted on the hardware front.

Virtualization shoot-out: World's fastest hypervisor
The test plan was straightforward: Take a look at Windows and Linux server performance on the physical hardware, then on an otherwise quiescent hypervisor, as well as several more times on a hypervisor under increasing load levels. Metrics included CPU, RAM, network, and storage I/O performance, time and interruption (if any) during VM migrations, speed and agility in VM template creation and deployment, and overall handling of a few disaster scenarios, such as the abrupt loss of a host and failover to an alternate site.

The benchmarks themselves were based on synthetic and real-world tests. They provide a general picture of hypervisor performance, but as with many facets of virtualization, there's no good way to accurately forecast how any workload will perform under any virtualization solution apart from running the actual workload.

The Linux tests were drawn from my standard suite of homegrown tests. They are based on common tools and scenarios, and they're measured by elapsed time to complete. These included converting a 150MB WAV file to MP3 using the LAME encoder on Linux, as well as using bzip2 and gzip to compress and decompress large files. These are single-threaded tests that are run in series, but with increasing concurrency, allowing performance to be measured with two, four, six, eight, and twelve concurrent test passes running. By running these tests on a virtual machine with four vCPUs (virtual CPUs), we were able to measure how well the hypervisor handled increasing workloads on the VM in terms of CPU, RAM, and I/O performance, as all files were read from and written to shared storage.

The Windows tests were run with SiSoftware's Sandra. We chose to focus on a few specific benchmarks, primarily based on CPU and RAM performance, but also including AES cryptography, which plays a significant part in many production workloads.

Again, all tests were conducted on the same physical hardware, with the same EqualLogic PS6010XV iSCSI array for storage, and on identical virtual machines built under each solution. All the Windows tests were run on Windows Server 2008 R2, and all the Linux tests were run on Red Hat Enterprise Linux 6 -- with the exception of Microsoft Hyper-V. Because Hyper-V does not support Red Hat Enterprise Linux 6, we used RHEL 5.5, which may have had a minor impact on Hyper-V's Linux test results.