Java unit tests you forgot to run

AgitarOne exercises Java code in unique and compelling ways, but not without some agitations for users

Unit testing -- a form of software testing done by developers using hundreds of small, fast tests -- is a central practice at sites that are committed to software quality. By following a dictum of "unit test, rather than debug," developers and their managers identify problems early and solve them as they go, giving them confidence in the code under development and assurance that a project will not founder once development completes and the quality assurance engineers start testing.

Agitar is a company whose entire product line is devoted to unit testing. Its flagship Agitator product (see our April 2006 review) reads through Java code, creates tests on the fly, runs them in the background, and then reports the results (successes and failures) as well as the lines of code that were not exercised. The benefit of this approach is that Agitator will often create tests that developers would overlook due to excess familiarity with the code. In addition, Agitator runs phone numbers, Social Security numbers, and other common fields through a gamut of extensive prebuilt tests.

AgitarOne, the focus of this review, is the heir to Agitator. During the next year, AgitarOne will replace Agitator at most customer sites, as Agitator is being retired. AgitarOne is a software stack that expands on Agitator's capabilities in useful and innovative ways. However, it is a heavyweight solution that may amount to more software than smaller teams need or want.

Click for larger view.

Dashboard for developers

AgitarOne follows a trend that has been emerging in software development tools during the past year, which is to provide a dashboard with project metrics. (In the past 12 months, weve seen products such as Enerjy's CQ2 and Borland/CodeGear's JBuilder 2007, among others, adopt this approach.) AgitarOne's dashboard is hosted on CruiseControl, an open source product that serves as a platform of sorts for generating and executing unit-tests, applying code rules, and displaying the metrics dashboard.

The developer triggers these activities using client-side software that must be installed in a Java IDE. At the moment, Agitar supports only Eclipse and Eclipse-based IDEs (such as CodeGear JBuilder 2007 and IBM Rationals family of IDEs). This installation is done via a series of plug-ins that can be downloaded into Eclipse from the URL of the AgitarOne server. Some activities can be run in a command-line batch mode.

Once everything is set up, the developer clicks on a project or a file, and from an Agitar-specific, drop-down menu elects whether to generate unit tests, "agitate" the code, or run the code-rules engine. A queue can then be checked to watch the progress of these tasks.

The unit tests AgitarOne generates have two benefits: They can fill in test suites written by the developer, and they can capture current behavior. To its credit, Agitar is clear that its generated tests are a supplement to developer-created tests, not a replacement for them. The first of these benefits is most important where unit tests are used as a quality-validation tool. AgitarOne can create tests the developer would not have thought of and catch errors by exercising the code in legitimate ways the developer did not foresee, since a frequent blind spot for developers is their over-familiarity with code they've written. AgitarOne's generated tests do this well. In this sense, they are instructive, and regularly examining them will surely help most developers broaden their own test-writing skills. The second benefit of generated tests is capturing existing behavior so that when a developer changes the code, failing tests can highlight unanticipated consequences. This usage, termed "characterization tests," is particularly valuable when a developer is modifying legacy applications.

The agitation option runs many more tests in the background. The additional tests are created on the fly, often throwing random values at variables and functions. It is a purely quality-oriented activity. When a test fails, AgitarOne reports the sequence of functions and the values passed to them so that the developer can trace through the code to see why the failing value was not handled as expected.

Click for larger view.
The code-rules module is a stripped-down, static code analyzer that looks for violations of coding standards and a few suspect constructions. It relies largely on the open source Checkstyle tool to do most of the verifying, although Agitar has added some suggested rules of its own. The results of the scan (and those of agitation and testing) are reported in Eclipse and also on the dashboard hosted on the AgitarOne server.

One step forward and back

AgitarOne is a unique product; however, it needs to mature some more before it can be freely recommended. Installing the product created curious difficulties that were tough even for Agitar engineers to diagnose. In addition, running AgitarOne and exploiting its features was made unnecessarily hard by poor (but copious) documentantion marred by invalid URLs, mistaken names for files, procedures with missing steps, and other gaps. Given the unique nature of the product, this carelessness can become a showstopper, because there is no "common knowledge" users can rely on to find their way.

In addition, the product had a surprising brittleness to it, given that Agitar has been selling its core technology for years. The server unaccountably would become unavailable or lose HTML pages. When files or configurations were not as expected, the software would exhibit unusual symptoms that were at times far afield from the originating cause, making correction unnecessarily difficult. An upcoming release is expected to address many of these issues.

Finally, there are a few limitations that users should recognize: to use AgitarOne you must use Eclipse, an unattractive option to some developers. In addition, the product generates unit tests only for JUnit 3.8.1. JUnit 4 and other unit-testing frameworks are not supported, nor are languages other than Java.

These caveats aside, AgitarOne is an easy-to-use workgroup product that greatly facilitates the use of unit tests and helps sites get as much benefit as possible from this activity. The result is shorter QA and debugging cycles and much better predictability of the software process. For many sites with large, important Java projects, this solution is attractive and compelling. All such sites are likely to derive value that far exceeds the cost.

Existing Agitar customers will eventually be forced to migrate, as the company is phasing out its Agitator product. Those customers will like what they find, especially the unit-test generation and the dashboard, but they should allow the product to mature some before beginning the transition.

InfoWorld Scorecard
Ease of use (20.0%)
Performance (15.0%)
Scalability (15.0%)
Capability (30.0%)
Value (10.0%)
Documentation (10.0%)
Overall Score (100%)
AgitarOne 4.1.1 8.0 8.0 8.0 9.0 8.0 6.0 8.1