Study overstates improvements in open source code quality

Coverity's data supports claims of a 16 percent improvement or a 21 percent decline in open source code quality, depending on how the results are calculated

Data from the Coverity Scan Open Source Report found a 16 percent improvement in the quality of open source projects actively participating in the scan. This is exactly the type of data that open source vendors and proponents want in their back pockets. But is it accurate? Hold your tomatoes and let me explain.

Coverity used lines of code (LOC) before a defect was found to evaluate code quality and code quality improvements in a normalized fashion. There are, however, two approaches for doing this calculation.

[ Samba, tor, OpenPAM, and Ruby were granted top-level status for resolving defects in Coverity's study. | Stay up to speed with the open source community via InfoWorld's Technology: Open Source newsletter. ]

The first approach is to calculate the total LOC scanned across all projects and divide by the total number of defects found across all projects. The result is the average number of LOC before a defect is identified. This one-step approach is best for talking about the code quality of the overall code base, across projects. It does not, however, let us determine which projects are producing higher-quality code versus other projects or track results on a project-by-project basis over time.

The second approach is a two-step process. First, calculate the LOC per defect on a project-by-project basis. Then average those results to arrive at the LOC per defect across all the projects scanned. The first step of this approach is good for understanding which projects are producing higher-quality code and measuring a project's progress in the quality arena over time. However, averaging the LOC per defect across projects in step two disregards the size of a project's code base.

Before we move on, keep in mind that a higher LOC per defect number is preferable regardless of which approach we use. For example, a 999-LOC-per-defect result is better than a 100 LOC per defect.

As I'm sure you'd expect, or I wouldn't be blogging about this, the two approaches provide different results due to how the code base size of a given project is weighted. In approach one, projects with larger code bases are weighted higher than projects with smaller code bases. The reverse is true for approach two.

Here's an example:

Project A: 2,000,000 LOC & 2,400 defects
Project B: 20,000 LOC & 20 defects
Project C: 21,000 LOC & 15 defects

Using the first approach, the LOC per defect would be: 838