Quest brings sweet simplicity to database benchmarking
Benchmark Factory 4.7.1 is a snap to use but falls short on reporting
The one bittersweet addition to this version of Benchmark Factory is the goal-based testing. This feature allows you to run a benchmark that automatically increases the user load until a certain condition is met, rather than running a timed benchmark and interpreting the results. Goal-based testing can be very useful when you’re trying to find the maximum capabilities of a system because it keeps you from having to run several benchmarks to determine the exact load the server can handle.
That’s the sweet part. The bitter part is that you can’t use goal-based testing to diagnose possible performance bottlenecks; you are limited to TPS (transactions per second) and response time. This limitation is fine if you live in the world of commercial benchmarking, but for real-world benchmarking, DBAs need to measure real results, not simply TPS. They need benchmarks that will increase the user load until the server reaches a certain sustained disk queue, or a certain percentage of CPU, or memory, or any other stress condition they may be testing for.
After performing a benchmark, the most important thing is interpreting the results. Benchmark Factory makes it very easy. For starters, you can specify any of the Windows performance counters, and the graphs will be available when the benchmark has been completed. The reports themselves are quite detailed and can be viewed for various aspects, such as CPU usage vs. user load. You also can generate real-time graphs. Currently the reports can be exported only to Excel, but that’s probably going to be the tool of choice anyway.
The major shortcoming in the reporting model is that you can’t create reports automatically to compare various counters side by side. You have to generate the reports separately and then merge them manually into a graph in Excel (or another tool). You also can’t pivot the charts to view data elements against one another. A good example would be comparing disk queues against the number of users, instead of disk queues vs. number of active transactions.
Also, the reports that Benchmark Factory generates are basic and lack polish. It would be nice to see Quest employ SSRS (SQL Server Reporting Services) or Crystal Reports as a reporting engine in future releases so that DBAs could get some quality reports they could send to management out of the box.
Quest Benchmark Factory is finally becoming an enterprise-level benchmarking tool. It has an intuitive interface that will have even fairly novice users up and running with few problems. The reporting is adequate, but the inability to produce pivot reports creates more work for users. Finally, the various client agents allow you to set up complicated load scenarios, but you can only run one test at a time from a single console.
Read more about data management in InfoWorld's Data Management Channel.