Debunking benchmarks

At InfoWorld, we do battle with product testing every day, both within the Test Center and within InfoWorld's own IT group. What paces do you put a system through before you decide that it is ready for prime-time? President Eisenhower once said, "In preparing for battle, I have always found that plans are useless, but planning is indispensable." As a CTO, I'd rephrase that quote to say, "When dealing with real-world IT issues, I have always found that benchmarks are useless, but testing is indispensable."

When you talk about "testing," the word "benchmark" usually does come up, but benchmarks really are only part of the story — a very small part. Think of benchmarks as putting a car on rollers to see how high you can rev the speedometer: The wheels might spin at 130 mph, but what happens on a real road when there's rain on the ground and unpredictable wind? Besides, are you going to be driving that Toyota Camry 130 mph around town to conduct your regular business, or do you have more important requirements than pure speed? You're probably looking at gas mileage, maintenance costs, and whether or not that Camry fits your lifestyle.

Similarly, if you're testing enterprise software and hardware, you're not running a race. Yet benchmarks typically treat IT testing as a race of absolutes in which someone can "win." As computing power gets cheaper, the qualitative experience of installing, configuring, and managing software becomes much more important in determining a "winner" in your particular environment.

From a pure speed standpoint, most products will perform pretty much as advertised. Can you guess what the throughput of a Intel gigabit switch is vs. a Cisco gigabit switch? To be honest, it probably doesn't really matter from a performance standpoint — both are really fast. For most applications of a gigabit switch in a corporate environment, either switch will function just fine. But you'll have to set it up first and integrate it into your environment — that's the hard part.

Your typical LAN user will not complain about millisecond differences in file transfers. The more important issue is the management of the switch and if it plays well with others. Your sales staff might cut you some slack for losing a millisecond here and there, but they won't give you a break if the network is down because the switch you chose has obtuse management features that you failed to notice while watching your stopwatch to see which switch pushed through 10,000 64-byte packets the fastest. The fastest switch might win a benchmark, but it's you who will lose in this case, as you sit with the switch manual hunting for that arcane configuration tweak while the network grinds to a halt. Blinding speed can be, well … blinding.

During the past 18 months at InfoWorld, we have implemented a new content management system for our Web site, migrated most of our back-end Web site systems from Solaris to Linux, migrated from Netware to Windows 2000 for LAN file-and-print services, rolled out wireless e-mail solutions on top of our legacy Lotus Notes environment, and implemented an anti-spam solution for InfoWorld's employees. In some cases, we ran some benchmarks. But guess what? The benchmarks we ran usually proved our assumptions correct, and it wasn't until we got the systems out into the "real world" of end-users that we understood the true limitations of the systems. No benchmark could have revealed that a salesperson might have trouble navigating the interface of an SFA system.

IT is about real people and real end users. That's an environment benchmarks will never touch.