Quest brings sweet simplicity to database benchmarking

As applications become more and more complicated, and businesses grow more reliant on electronic transactions, benchmarking becomes increasingly important. Businesses could really never get along without benchmarking, and some are starting to feel the pains of neglecting this process as their business needs grow.

The truth is that there aren’t a lot of database benchmarking utilities out there, and the ones that do exist have traditionally been script-based and rather complicated to understand, configure, and operate. Well, all that has changed with Quest Benchmark Factory 4.7.1.

Quest Software has actually slimmed down this offering, reducing it from a full-service benchmarking utility covering applications, file systems, and e-mail, to one covering only databases. Nevertheless, the latest version has plenty to offer and is extremely easy to use. It’s not without its shortcomings, however. Reporting capabilities, in particular, could be improved.

One of my favorite features is Benchmark Factory’s host of pre-built standard benchmarks. With just a few clicks, you can load a standard TPC-C (Transaction Processing Performance Council Benchmark C), choose from a variety of other predefined tests, or create your own. Configuring and running these benchmarks is a snap; creating all the necessary objects is just as easy.

The wizard’s most useful feature is the Data Exploder, with which you can generate as much test data as you need to run a test. It’s so simple: You don’t even need to know anything about any of the tables or their relationships. All you have to do is specify the size you want the database to be, and Benchmark Factory automatically creates all of the data you need with all the correct proportions.

Using this generic form of benchmarking is great when you need to test the capabilities of your server hardware. Benchmark Factory allows you to configure benchmarks that concentrate on the various areas of server performance, such as disk, CPU, and memory. You can’t alter the predefined tests in any way, however, nor does the solution give you access to the SQL statements being run.

Creating your own benchmarks from scratch is incredibly simple, too. After you have chosen the type of scenario you want to run (mixed or replay), you just start adding transactions and choosing their weights. When adding a transaction, you can set up dynamic execution variables by choosing various variable types inside the scripting wizard.

There are serious limitations to using the dynamic variables, however. Benchmark Factory can’t perform lookups of your actual data to use as dynamic parameters, and it lacks support for data types such as GUIDs (globally unique identifiers), decimals, and floats. Even if you’re able to build in that support yourself, the selections would be so random that you may not get enough hits on the data to achieve a reliable benchmark score. You can, however, choose random values, as well as ranges of random values.

For most benchmarks, I found the parameter facilities adequate, but Benchmark Factory didn’t cross the finish line for complicated scenarios or even simple scenarios with more sophisticated data requirements.