Vendor benchmarks: Buyers beware
Recent SPEC benchmark results show EMC's new storage array trouncing the competition, but it's tough to gauge what pushed its wares to the front of the pack
Follow @tsamson_IW
Third-party benchmarks from organizations such as SPEC are a great tool for companies shopping around for new IT gear -- in theory. They could provide an unbiased, apples-to-apples comparison of results from identical tests to determine which server or storage array is the best for particular workloads. Not surprisingly, though, the vendors typically don't use them that way.
For example, a casual glance at recently published SPEC benchmark results would suggest that EMC's new VNX storage array is so darn fast that competitors such as NetApp and HP may as well pack up their file servers and go home. But closer scrutiny reveals that although EMC may indeed be pushing the envelope on file-server speeds, companies in the market for new storage gear still have comparison shopping to do.
What's behind the numbers?
According to recent SPECsfs2008_cifs results comparing the performance of storage systems running a CIFS workload, EMC's VNX storage array achieved throughput of 661,951 ops (operations per second) with an ORT (overall response time) of 0.81 millisecond. The next-best results on the list from a non-EMC competitor came from NetApp: Its FAS3140 system, running FCAL disks, managed a throughput of 55,476 ops and an ORT of 1.84 ms. Given only those figures, one might assume that EMC come up with some extraordinarily potent secret storage sauce that will ease Big Data pains worldwide.
But closer inspection suggests there's more than uber EMC technology at play: For starters, EMC ran its test on a VNX VG8 Gateway/EMC VNX5700, with five X-Blades (including one standby). The configuration is composed of 581 disks, 240GB of memory, and a 10GBE network.
By contrast, no other non-EMC array on the list was tested on a 10GBE network; rather, most ran on a variant of 1 GBE. Additionally, the aforementioned NetApp system was configured with 224 disks (less than half the number of the EMC setup) and 9GB of memory.
Thus, the only clear takeaway from these results is that a system with lots of disks and memory running on a network with obese pipes is faster than a system with a modest amount of disks and memory running on a more traditional network.
