SAN FRANCISCO (08/11/2008) - The agile revolution that began in software development in the 1990s has been inexorably making its way into mainstream IT organizations. Today, one of the most adopted agile practices is unit testing, where developers write hundreds of small tests for exercising their own code. Although the benefits of unit testing are widely recognized, there's growing evidence that unit testing might have reached its high-water mark and be entering a period of stagnation or even decline.
Signs that unit testing has hit a wall include the high-profile collapse of Agitar, which was known widely for a good, if expensive unit-testing product. CEO Jerry Rudisin told the SD Times that Agitar's collapse (it was bought by McCabe Software in June) was due simply to the fact that "unit testing hasn't taken off as a mainstream practice." Joe Ponczak, who heads up Condign Software, the makers of automated unit test generation tools and metrics dashboards, says he believes the Agitar failure was brought on by slower-than-desired growth, "not at the level expected from a Silicon Valley startup."
Andrew Glover, president of the agile-development consulting firm Stelligent, blamed Agitar's high price. "Only very large enterprises could afford it. And it was hard to justify paying those prices for a product that simply automated what developers could already do for themselves with free tools," he says.
Why unit testing may be losing its allure
Of course, Agitar is just one company, and its experience is hard to translate as an overall trend -- especially because the major tools used in unit testing are available as free open source, such as JUnit and TestNG for Java and the xUnit frameworks for other languages, as well as the free code-coverage tools Cobertura and Emma. Most of the commercial products focus on automatic generation of unit tests and coverage analysis.
But consider the typical experience of Stelligent's Glover when he consults on development approaches with clients: "When I go into sites, I would love to turn them on to the benefits of the TestNG test framework," he says, "but most organizations do so little unit testing, if any, that I stick with the simple, well-known xUnit tools, so as not to create confusion. Most organizations are just not ready to learn anything more than basic unit testing -- if that."
While languages such as Python and Ruby have active unit-testing cultures, there are signs that, in the Java community at least, resistance has built up against the unit testing idea. I posted the question on unit testing's future in my blog recently, and got several hundred responses. Among those, I identified two major threads of concern.
One issue is that unit testing seems to offer little value. "The problem with unit testing is that it increases the cost of development before you know whether what you developed is what you needed. Afterwards when you have a working solution, there's little interest in writing unit tests as what you have works, and your management wants you to move on," wrote one anonymous poster. "Many developers and managers mistakenly think that by cutting testing, they well get feature X out faster. What they don't realize is that you are robbing Peter to pay Paul," wrote Peter B.
The other issue was annoyance over evangelism, which is at the core of much of the agile movement (which began with a signed manifesto and a formulated creed). "Please, please, please, no more evangelism!!" wrote Gabriel C. A segment of the agile development community has adopted unit testing in a radical way, insisting that tests be written before the code.
This approach, called test-driven development, is intended to clarify design before committing to code. Its proponents have been zealous in promoting test-driven development as the best way to program, but for many developers that evangelism has become so overbearing that it makes unit testing as a whole unappealing.