Load testers prove a valuable part of your development strategy
High-performance Web sites and apps don't happen accidentally. A detailed performance and scalability testing strategy is needed throughout the development lifecycle -- and that requires a good load-testing solution.
The rewards are clear: Strong, ongoing emphasis on performance testing can keep budgets in check and reduce production issues. With the plethora of performance and scalability tools available today, however, choosing the right one for your shop can prove challenging.
I recently examined three Web load-testing solutions: OpenSTA 1.4.3, Ingenieurbüro David Fischer's Proxy Sniffer Professional Edition 3.7, and Minq's PureLoad Enterprise Edition 3.3.1. The three solutions represent the low-, middle-, and higher-end price points of the load-testing tools market; and for basic HTTP and HTTPS testing, any of these three solutions more than meets the need. The way each solution is implemented differs, however, as does the functionality they offer.
OpenSTA is an open source Web load-testing tool that's easy to install and use. Setting up baseline and scalability types of load tests using either a single or multiple machines is a snap. Platform support, however, is surprisingly limited to Windows only.
Nonetheless, after registering all of my test machines with an OpenSTA repository on a central machine, I was executing distributed load tests in no time. Heavy OpenSTA load tests may also require developers to adjust the maximum number of sockets on each load client to reduce the occurrence of socket errors.
Script creation, execution, and analysis are all accessible via OpenSTA's Commander interface, which is a native Windows application. After creating a new script definition and clicking on it, the OpenSTA Script Modeler opened. The Script Modeler has three panes: one for script creation; a diagnostic window for script compilation and playback; and a third view that allows developers to see the HTML, site structure, and header information.
OpenSTA scripts are written using SCL (Script Control Language) -- a built-in compiled scripting language that has influences from Fortran and Digital's Digital Command Language. Reference documentation is provided to get developers going. Developers also might find it easy to learn SCL by first using the built-in capture facility.
After capturing interactive browser sessions with several test Web sites and apps, I read through the code and began quickly to grasp the language constructs. For most basic testing, you can simply capture, compile, and execute the script using the Commander.
For more productionlike load tests, you can use a variety of variable-based inputs. For example, I used ODBC to provide script input of customer information that I had on hand in a database. Likewise, you can input data into scripts from variables, a list of values, and files, such as Excel spreadsheets.
OpenSTA offers an impressive amount of customization and control over various portions of the test configuration. With scripts in hand, I returned to Commander and began defining tests. The test setup screen is a spreadsheetlike interface.
To set up the test, developers drag and drop scripts onto the testing grid. Each row in the testing grid depicts a task group, and each column depicts a task. A single task group can execute as many as 200 tasks, and developers can define as many as 200 task groups. This level of granularity can enable precise testing of the various portions of a Web site or app under varied conditions.
Within the testing grid, you can also define the number of iterations for a task, task group, or both. Moreover, you can define metrics, such as startup delays, think times (end-user idle time), and which machines should run which tasks. For example, in one of my tests I defined 20 task groups and had each of five load-test clients executing four of the task groups.
After the tests have been defined, they are started from the Commander interface. Switching to the monitoring tab, I could see basic data about the running tests, such as the number of virtual users and the number of successful and failed test calls. After the test is completed, developers can switch to the results tab to view reporting data.
OpenSTA offers basic built-in reports, such as errors versus the number of requests, and there is some drill down provided. OpenSTA does not, however, include a test-comparison function as do the other two tools evaluated here. But you can export data (CSV format) or data and charts to Excel for the purpose of comparison.
With the right price tag -- free -- OpenSTA is a solid, basic Web technology test tool that belongs in every IT toolbox. It may not have all of the bells and whistles of its counterparts, but it makes fast and accurate Web load testing a simple affair.
Proxy Sniffer Professional Edition 3.7
Of the three tools tested here, Proxy Sniffer gets the nod for offering a full-featured Web load-testing solution at an attractive price. It goes further than OpenSTA and offers most of the same Web load-testing capabilities as the more expensive PureLoad.
Proxy Sniffer is a Web browser-based solution that supports stand-alone and distributed testing leveraging Unix, Linux, and Windows load-test clients. The Proxy Sniffer server and GUI are graphically launched on Windows or are started via the command line on Unix and Linux.
Proxy Sniffer functionality is served up via a series of different browser windows, which launch from one another. In the initial browser window, I recorded several scripts and edited them. I was able to insert think times and add variables and the like. Proxy Sniffer scripts can use variable input from files or other variables to simulate end-user interaction during tests.
After they have been saved, the scripts are then accessible in the Project Navigator browser window. The next step is to generate a load test. To do this, Proxy Sniffer constructs a Java app on the fly. After it has been saved and compiled, the load-test application is available within the Project Navigator.
Developers can then define the parameters for load-test execution. In my tests I defined remote-agent execution using Unix and Linux machines and parameters, such as the number of virtual users and the length of the test run.
After my tests had been started, I was able to connect to and monitor them in yet another browser session. During the run time, testers can view summary and detailed statistics. When executing tests on the local network or across multiple proxies, Proxy Sniffer proved easy to configure and run.
At the conclusion of the tests, a simple click on a button acquired the output from the test into the analysis component of Proxy Sniffer. Particularly impressive are Proxy Sniffer's reporting and comparison facilities. Of the three tools tested here, it offers the most detailed reporting capabilities. I particularly liked its PDF output and comparison facilities. I simply entered a file name and generated a PDF that contained detailed data and charts from the test results.
After running several iterations of load tests to simulate baseline load plus successive business growth for three- and six-month intervals, I selected multiple test results, chose the comparison type (load curve or bar charts), and clicked the compare button.
It became quite clear that by the six-month interval, I would need to add servers to support the intended load, as the response time would have gone beyond the test SLA that I had previously defined. The compare charts and detailed drill down clearly showed the slowdown in response time with increased load. I successfully exported the comparison in HTML form.
Of the three solutions tested here, Proxy Sniffer offers the best ratio of features to price. In particular, its cross-platform support, ease-of-use, and in-depth reporting make it a strong contender and well worth short-list consideration if you are evaluating Web load-testing tools.
PureLoad Enterprise Edition 3.3.1
When compared with OpenSTA and Proxy Sniffer, PureLoad Enterprise Edition is capable of handling a broader range of testing duties. It is fully capable of testing Web sites and Web applications, but it also can be used to load test other enterprise technologies, such as JDBC connections, SMTP activities, and LDAP interactions.
A Java-based application, PureLoad is available in both a Web Edition (HTTP and HTTPS load testing) and an Enterprise Edition (Web load testing plus enterprise application load testing). If you need a single tool to gauge the performance and scalability of several types of technologies, PureLoad is a solid choice and should be on the short list.
When viewed through the lens of purely Web-based load-testing glasses, PureLoad is a tad pricey, though you get a lot for the money.
As does rival Proxy Sniffer, PureLoad supports single-machine as well as distributed load testing using Unix-, Linux-, and Windows-based load clients. For this review, I focused on how well PureLoad's Web technology testing facilities compared with that of rivals OpenSTA and Proxy Sniffer.
After starting PureLoad's services as outlined in the documentation, the next step is to launch the product's console. Defining a load test is very easy with PureLoad. After defining workers (virtual users), I then went on to create a series of scenarios (scripts).
PureLoad offers three very strong options for script creation. Test developers can choose to manually add tasks, such as an HTTP post task, to a scenario from the 70 built-in functions, or they can create custom tasks.
The second method of script creation would be to use PureLoad's Web Crawler tool, which crawls a Web site or application for static content and structure. This information can then be saved or copied into a scenario and be used as a foundation for a load test. The third method of script creation is provided by an included HTTP recorder, which captures interactive browser-session activity as the basis for a load test.
I found that each approach was solid and the different methods were complementary to one another. For example, after recording and executing a script, I was able to identify one area of my test Web site that was performing slower than expected under load. Using manual script creation, I was then able to focus a second round of testing on just that portion of the site.
Likewise, after defining the test site URL and the depth to scan in the Web Crawler utility, PureLoad accurately captured the content and structure of my site. I was then able to add some tasks manually to provide variable input during a load test from a file.
During the tests, PureLoad provided data and charts as it recorded the results of the execution. I was able to view overall metrics, such as successful and failed tasks and response time data, or drill down to the individual virtual user level to view the same types of metrics. At the conclusion of the test, the same metrics were then available for review and exporting.
Overall, PureLoad's reporting metrics are not as detailed as those found in Proxy Sniffer, but most testers should find them more than adequate.
PureLoad includes a load-test comparison tool, and after comparing the output of several load test runs, I found it to be on par with the comparison functionality provided by Proxy Sniffer. I also found it useful to export test data from PureLoad to a series of spreadsheets (CSV) and chart images (PNG) to perform further trend analysis.
Sites considering adoption of load-testing tools should evaluate PureLoad. In particular, its strong scripting facilities and added capability to support performance testing of both Web and enterprise apps make it appealing.
Any of the three tools tested here would be a useful addition to the toolboxes of enterprise performance-testing teams. OpenSTA provides solid core of Web load-testing tools at a budget-minded price, whereas PureLoad is an ideal fit for load testing Web and other enterprise technologies. Teams that need a full-featured Web load-testing tool at a reasonable price should definitely give Proxy Sniffer a test drive.
Ease of use (20.0%)
Developer tools (10.0%)
Overall Score (100%)
|Ingenieurbüro David Fischer Proxy Sniffer Professional Edition 3.7||9.0||9.0||9.0||9.0||9.0||9.0|
|Minq PureLoad Enterprise Edition 3.3.1||9.0||9.0||8.0||8.0||9.0||9.0|
Windows 7 is suddenly telling users it isn't genuine -- and it has nothing to do with Windows being...
Windows users are reporting significant problems with four more October Black Tuesday patches
The larger design is very welcome, but there's much more to the iPhone 6 than a bigger screen
Sponsored by Rackspace
Sponsored by Nuage Networks
Sponsored by Fibre Channel Industry Association
These legendary clunkers made Patch Tuesday a living hell for Windows users the world over
Git made it possible for programmers to coordinate distributed work across teams -- now GitHub makes it...
A new worldwide survey of mobile app developers reveals some interesting facts about their work, from...
High demand, large workloads, and the changing nature of programming work have some developers seeking...