Benchmark Chicanery

Idera has issued a press release outlining their new world record backup benchmark. As it stands, the benchmark results are reported in a misleading fashion, so let me go through some of this for you. "Using 'off the shelf' technologies available today to any data center customer, the companies achieved backup speeds in excess of 4.5 terabytes per hour and restores speeds in excess of 2.3 terabytes per hour. The

Idera has issued a press release outlining their new world record backup benchmark. As it stands, the benchmark results are reported in a misleading fashion, so let me go through some of this for you.

"Using 'off the shelf' technologies available today to any data center customer, the companies achieved backup speeds in excess of 4.5 terabytes per hour and restores speeds in excess of 2.3 terabytes per hour. These metrics reflect a more than 20% and 75% performance increase, respectively, over previously documented benchmarks for SQL Server backup and recovery."

OK, that's directly out of the press release and I think it outlines a couple really important parts. First, look at the backup speeds... 4.5 TB/hr. Man, that's fast. In fact, it's a lot faster than any backup benchmark out there, so they're right about that. However, what did they do to make it so fast? I mean after all, I've never known them to be that much faster than LiteSpeed or Red Gate. The secret is found where they say that the hardware is open to any data center. It didn't take me long to chase it down after I read it just a little more closely.

What's going on here is they're using Solid State Disks (SSD) on their system. For those of you who don't know what SSDs are, they're really super fast disks that are basically huge banks of RAM instead of platters. Their speed is measured in microseconds instead of milliseconds found in SCSI. The bottom line is that SSDs are exponentially faster than SCSI, but the cost also reflects that as well. In fact, they're really just way too expensive for most shops to even consider.

While Idera isn't really being deceitful (because they are disclosing the hardware details), they are being deliberately sneaky by not telling us that SSDs are what's being used. In fact, they're using a 16-way 64-bit box with SSDs. That's just an insane system and it better damn well be faster than anything else out there. And sure, SSDs are available to anyone who wants to buy them, but who's going to do that?

The reason I say they're being sneaky is because absolutely nobody out there is using SSDs in production, and they didn't take any other backups to use as a baseline. If it were a responsible benchmark it would have listed the SQL Server native backup times and then the Idera backup times. Of course they beat everyone else's benchmarks, they're running on hardware like a gazillion times faster, but how does it do against the other vendors on the same hardware? Will it still show such incredible results? My guess is NO. Hell, you could probably put a single-threaded QBasic application on that system and still record a world record benchmark.

Doing the benchmark this way is the same as a car company saying that their car is the fastest on the road by going from 0 to 120 in 5secs. Of course, they used nitrous oxide, and special tires, but that part is completely irrelevant. The point is they have the world record for street cars.

OK, to keep this kinda short, I'm just going to bottom line this for you real quick. It's not really Idera's job to make sure you understand the details of their benchmarks. That's my job. Idera's job is to put nice big headlines up saying 'world record benchmark' so you'll see that and buy their code.

I speak out against this kind of chicanery when I can because I honestly believe it's wrong. If they wanted an honest benchmark that would stand the test of production, they would have used SCSI. Their hardware platform is something that nobody will be able to reproduce, so their results will never be reproduced either. DBAs are going to be wondering why they aren't seeing the awesome results they read in the benchmark.

So for the record there is no world record benchmark here. It's just a marketing device used to catch your eye and steer you away from the fact that Idera is about as fast as everyone else. The data and compression level also play a huge role in these types of benchmarks too. A while back I did a comparison benchmark between LiteSpeed and Idera and depending on the data I used, and the compression level, each one of them was able to beat the other one at different times. That's just the nature of the game. So, what's likely here is they're probably using something similar to SAP data, which is easily compressible, and they probably also used their lowest compression level. I don't blame them on that part. I know the LiteSpeed guys use their lowest compression level when they do their benchmarks too. The difference is they use hardware that people are actually using in production.

I'd still like to see this benchmark done on the same data set with LiteSpeed and Red Gate, and MS native.

Here's the hardware spec:

• Intel Itanium 2 Processors

NEC Express5800/1320Xe

Emulex LightPulse® 4Gb/s LP11000 HBAs

Texas Memory Systems RamSan-400

Idera SQLsafe v3.0

Microsoft SQL Server 2005

Windows 2003 Datacenter Edition

Read my book reviews at:

http://www.ITBookworm.com

Related:

Copyright © 2006 IDG Communications, Inc.

How to choose a low-code development platform