Java unit tests you forgot to run
AgitarOne exercises Java code in unique and compelling ways, but not without some agitations for users
The unit tests AgitarOne generates have two benefits: They can fill in test suites written by the developer, and they can capture current behavior. To its credit, Agitar is clear that its generated tests are a supplement to developer-created tests, not a replacement for them. The first of these benefits is most important where unit tests are used as a quality-validation tool. AgitarOne can create tests the developer would not have thought of and catch errors by exercising the code in legitimate ways the developer did not foresee, since a frequent blind spot for developers is their over-familiarity with code they've written. AgitarOne's generated tests do this well. In this sense, they are instructive, and regularly examining them will surely help most developers broaden their own test-writing skills. The second benefit of generated tests is capturing existing behavior so that when a developer changes the code, failing tests can highlight unanticipated consequences. This usage, termed "characterization tests," is particularly valuable when a developer is modifying legacy applications.
The agitation option runs many more tests in the background. The additional tests are created on the fly, often throwing random values at variables and functions. It is a purely quality-oriented activity. When a test fails, AgitarOne reports the sequence of functions and the values passed to them so that the developer can trace through the code to see why the failing value was not handled as expected.
Click for larger view.
One step forward and back
AgitarOne is a unique product; however, it needs to mature some more before it can be freely recommended. Installing the product created curious difficulties that were tough even for Agitar engineers to diagnose. In addition, running AgitarOne and exploiting its features was made unnecessarily hard by poor (but copious) documentantion marred by invalid URLs, mistaken names for files, procedures with missing steps, and other gaps. Given the unique nature of the product, this carelessness can become a showstopper, because there is no "common knowledge" users can rely on to find their way.
In addition, the product had a surprising brittleness to it, given that Agitar has been selling its core technology for years. The server unaccountably would become unavailable or lose HTML pages. When files or configurations were not as expected, the software would exhibit unusual symptoms that were at times far afield from the originating cause, making correction unnecessarily difficult. An upcoming release is expected to address many of these issues.
Finally, there are a few limitations that users should recognize: to use AgitarOne you must use Eclipse, an unattractive option to some developers. In addition, the product generates unit tests only for JUnit 3.8.1. JUnit 4 and other unit-testing frameworks are not supported, nor are languages other than Java.
These caveats aside, AgitarOne is an easy-to-use workgroup product that greatly facilitates the use of unit tests and helps sites get as much benefit as possible from this activity. The result is shorter QA and debugging cycles and much better predictability of the software process. For many sites with large, important Java projects, this solution is attractive and compelling. All such sites are likely to derive value that far exceeds the cost.
Existing Agitar customers will eventually be forced to migrate, as the company is phasing out its Agitator product. Those customers will like what they find, especially the unit-test generation and the dashboard, but they should allow the product to mature some before beginning the transition.