The SPECsfs2008_nfs.v3 results, which compare performance running NFSv3 workloads, provide slightly more comparable test beds. Still, it's not quite an apples-to-apples comparison.
EMC, HP, and NetApp all listed systems tested on a 10GBE network. EMC's aforementioned VG8 Gateway/EMC VNX5700 combo, with 457 disks and 240GB of memory, achieved a throughput of 497,623 ops with an ORT of 0.96 ms.
By comparison, HP's best score came from its BL860c i2 4-node HA-NFS cluster, loaded with a 1,480 disks and 800GB of memory. That arrangement yielded a throughput of 333,574 ops and an ORT of 1.68 ms.
NetApp, meanwhile, fared best its FAS6240, equipped with 288 disks and 1,128GB of memory. That system managed a throughput of 190,675 ops at an ORT of 1.17 ms.
In this contest, it's tough to attribute EMC's results to an abundance of extra disks, memory, and fat pipes. HP brought far more of each to the party and still fell short compared to EMC. NetApp's entry had fewer disks -- but far more memory -- and its overall scores were considerably lower than EMC's.
Even with somewhat closer system configurations, the picture isn't fully clear. For example, one would certainly want to know how much money each of these setups would cost to purchase and operate, how they might run on a 1GBE network, what features they respectively bring to the table, and so on. Still, these benchmark results might help an IT shop come up with a short -- or shorter -- list of potential candidates for consideration.
Vendors can and will game benchmarks and cherry-pick results. There's no way around it. Still, benchmarks such as SPEC do provide some level of transparency -- particularly compared to secretive "in-house tests" or benchmarks performed by suspicious third parties -- not to mention side-by-side (though not necessarily apples-to-apples) comparisons among competing products. IT buyers just need to remain vigilant when digesting benchmark results.
This story, "Vendor benchmarks: Buyers beware," was originally published at InfoWorld.com. Get the first word on what the important tech news really means with the InfoWorld Tech Watch blog. For the latest developments in business technology news, follow InfoWorld.com on Twitter.