Quest brings sweet simplicity to database benchmarking

Benchmark Factory 4.7.1 is a snap to use but falls short on reporting

As applications become more and more complicated, and businesses grow more reliant on electronic transactions, benchmarking becomes increasingly important. Businesses could really never get along without benchmarking, and some are starting to feel the pains of neglecting this process as their business needs grow.

The truth is that there aren’t a lot of database benchmarking utilities out there, and the ones that do exist have traditionally been script-based and rather complicated to understand, configure, and operate. Well, all that has changed with Quest Benchmark Factory 4.7.1.

Quest Software has actually slimmed down this offering, reducing it from a full-service benchmarking utility covering applications, file systems, and e-mail, to one covering only databases. Nevertheless, the latest version has plenty to offer and is extremely easy to use. It’s not without its shortcomings, however. Reporting capabilities, in particular, could be improved.

One of my favorite features is Benchmark Factory’s host of pre-built standard benchmarks. With just a few clicks, you can load a standard TPC-C (Transaction Processing Performance Council Benchmark C), choose from a variety of other predefined tests, or create your own. Configuring and running these benchmarks is a snap; creating all the necessary objects is just as easy.

The wizard’s most useful feature is the Data Exploder, with which you can generate as much test data as you need to run a test. It’s so simple: You don’t even need to know anything about any of the tables or their relationships. All you have to do is specify the size you want the database to be, and Benchmark Factory automatically creates all of the data you need with all the correct proportions.

Using this generic form of benchmarking is great when you need to test the capabilities of your server hardware. Benchmark Factory allows you to configure benchmarks that concentrate on the various areas of server performance, such as disk, CPU, and memory. You can’t alter the predefined tests in any way, however, nor does the solution give you access to the SQL statements being run.

Creating your own benchmarks from scratch is incredibly simple, too. After you have chosen the type of scenario you want to run (mixed or replay), you just start adding transactions and choosing their weights. When adding a transaction, you can set up dynamic execution variables by choosing various variable types inside the scripting wizard.

There are serious limitations to using the dynamic variables, however. Benchmark Factory can’t perform lookups of your actual data to use as dynamic parameters, and it lacks support for data types such as GUIDs (globally unique identifiers), decimals, and floats. Even if you’re able to build in that support yourself, the selections would be so random that you may not get enough hits on the data to achieve a reliable benchmark score. You can, however, choose random values, as well as ranges of random values.

For most benchmarks, I found the parameter facilities adequate, but Benchmark Factory didn’t cross the finish line for complicated scenarios or even simple scenarios with more sophisticated data requirements.

The one bittersweet addition to this version of Benchmark Factory is the goal-based testing. This feature allows you to run a benchmark that automatically increases the user load until a certain condition is met, rather than running a timed benchmark and interpreting the results. Goal-based testing can be very useful when you’re trying to find the maximum capabilities of a system because it keeps you from having to run several benchmarks to determine the exact load the server can handle.

That’s the sweet part. The bitter part is that you can’t use goal-based testing to diagnose possible performance bottlenecks; you are limited to TPS (transactions per second) and response time. This limitation is fine if you live in the world of commercial benchmarking, but for real-world benchmarking, DBAs need to measure real results, not simply TPS. They need benchmarks that will increase the user load until the server reaches a certain sustained disk queue, or a certain percentage of CPU, or memory, or any other stress condition they may be testing for.

After performing a benchmark, the most important thing is interpreting the results. Benchmark Factory makes it very easy. For starters, you can specify any of the Windows performance counters, and the graphs will be available when the benchmark has been completed. The reports themselves are quite detailed and can be viewed for various aspects, such as CPU usage vs. user load. You also can generate real-time graphs. Currently the reports can be exported only to Excel, but that’s probably going to be the tool of choice anyway.

The major shortcoming in the reporting model is that you can’t create reports automatically to compare various counters side by side. You have to generate the reports separately and then merge them manually into a graph in Excel (or another tool). You also can’t pivot the charts to view data elements against one another. A good example would be comparing disk queues against the number of users, instead of disk queues vs. number of active transactions.

Also, the reports that Benchmark Factory generates are basic and lack polish. It would be nice to see Quest employ SSRS (SQL Server Reporting Services) or Crystal Reports as a reporting engine in future releases so that DBAs could get some quality reports they could send to management out of the box.

Quest Benchmark Factory is finally becoming an enterprise-level benchmarking tool. It has an intuitive interface that will have even fairly novice users up and running with few problems. The reporting is adequate, but the inability to produce pivot reports creates more work for users. Finally, the various client agents allow you to set up complicated load scenarios, but you can only run one test at a time from a single console.

InfoWorld Scorecard
Reliability (20.0%)
Performance (20.0%)
Reporting (15.0%)
Value (10.0%)
Setup (10.0%)
Manageability (25.0%)
Overall Score (100%)
Quest Benchmark Factory 4.7.1 9.0 8.0 6.0 8.0 9.0 8.0 8.0