Group to study virtualization benchmarks

A technology industry standards body is considering setting benchmarks for how virtualization should work in a computer network.

Free IT resource Virtualization Insights from Top Experts - Learn how virtualization gets real! Sponsored by Dell Free IT resource Try Sun servers, workstations and storage products free for 60-days. Sponsored by Sun Microsystems

The Standard Performance Evaluation Corp. (SPEC) this week is forming a group to discuss whether virtualization benchmarks are needed and, if so, then set those benchmarks over an estimated three-month process.

"It seems like a number of our member companies have simultaneously agreed they are ready to move on it and form a working group," said Walter Bays, president of SPEC "All of them must have been hearing the same thing from their customers."

SPEC is a nonprofit organization of top companies in the technology industry collaborating to establish impartial standards to measure the performance of various types of technology. SPEC has set benchmarks for servers, software applications, high performance computing systems, for example.

Virtualization is a growing field for improving the performance of data centers without having to add more servers. For instance, virtualization makes it possible to run several software applications, even based on different operating systems, on the same physical server.

Because many IT managers at companies are still learning what virtualization is, benchmarks would help educate them, said Bays.

"It would actually increase the market for virtualization because it would reduce uncertainty about it for the buyers," he said. "They would be better able to do comparison shopping."

The virtualization market doesn't seem to be hindered by so far. VMware Inc., the subsidiary of storage vendor EMC Corp. that sells virtualization software, on Oct. 17 reported an 86 percent growth in third quarter revenue to US$188.5 million, compared to the third quarter of 2005. The third quarter revenue, up 20 percent from the second quarter, translates into an annual run rate of $750 million, VMware said.

"Based on the adoption rate, benchmarking is not first and foremost in customers' minds," said Brian Byun, vice president of global partners and solutions for VMware. Whether one virtualization software package runs 5 percent or 10 percent faster than another isn't as important as the increased performance of virtualization versus nonvirtualization, Byun said.

The lack of benchmarks isn't preventing Advanced Micro Devices Inc. from touting the virtualization capabilities of its microprocessors.

"Virtualization is getting deployed. It's beyond the test beds and R&D labs," said Patrick Patla, director of server and workstation marketing for AMD. Still, he added, "We would love to have an industry standard benchmark."

Benchmarks are also needed to compare different types of virtualization software. Although VMware is the leader, the benchmarking process would compare virtualization software from multiple vendors.

"We do believe that industry standards are very important but they should be vendor neutral," said Lorie Wigle, marketing director for server, software and technology initiatives for chip maker Intel Corp.

Intel already offers a virtualization measurement tool, called VConsolidate, which hardware vendors and customers use to gauge how virtualization will work in various computing environments.

A Web server, an e-mail server or a database program all have different performance benchmarks, but they could all operate simultaneously on one virtual server, so performance benchmarks have to be rethought.

"Virtualization requires a completely different usage model," Wigle said.

SPEC's Bays said the working group will include representatives from Intel, AMD, Fujitsu Siemens Computers, VMware and other major vendors, but that SPEC would also like to invite end users to participate in the process.