Ben Chelf on Reliability in Complex Software Systems

Last month when I was covering ESC I asked how we can guarantee the reliability of complex embedded software systems. Ben Chelf, CTO of Coverity just emailed me a response that I think you might find interesting: Martin, I noticed your recap on the ESC conference. I was in Boston presenting two topics, one at ESC on the application of next generation static analysis to embedded software and one at SD Best Practi

Last month when I was covering ESC I asked how we can guarantee the reliability of complex embedded software systems. Ben Chelf, CTO of Coverity just emailed me a response that I think you might find interesting:

Martin,

I noticed your recap on the ESC conference. I was in Boston presenting two topics, one at ESC on the application of next generation static analysis to embedded software and one at SD Best Practices on next generation software development tools. Quite a few of your comments resonated with me. You talk about the increasing complexity in automobiles in terms of number of control units. Another thing to think about is the software code that controls those units. In fact, there are more than 10 million lines of code on a premium automobile these days. The complexity of a system like that is mind-boggling, and certainly now it's impossible for any individual or even group of individuals to keep straight in their head all at once. That's why reliability is so hard right now and your question is dead on -- how do we guarantee reliability?

The good news is there is a reasonable model for the folks developing all this software. Conventional static and dynamic analysis have been attacking this problem with some success over the past few years, but the opportunity to innovate within this space is still there. For example, if we look to chip designers, we see there are tools for chip design that are based on rigorous verification and static analysis for detecting defects in the hardware designs. The price of failure in sending a bad design for physical fabrication is tremendous and therefore, the tools are necessarily very sophisticated. It's time for us to bring that sophisticated level of analysis into the software development world.

Static analysis for software is a concept that is not new. There have been companies that have provided static analysis solutions for a long time (most notably in a tool called Lint). But in the past, the biggest problem for all of these tools for software has been "false positives" -- where the tool reports something that just is not correct. Part of the reason for these false positives has to do with some fundamental limitations in the tools that followed Lint.

Eliminating false positives is no easy task because software systems are so much more complex than chips. Today, companies are using sophisticated new technology to perform Path Simulation statically through dataflow analysis to cover 100% of potential paths through the code, in addition to leveraging Boolean Satisfiability for analyzing code. Boolean Satisfiability is the same type of technology that Synopsys, Cadence, and other EDA tools providers leverage in verifying chip designs. This new technology, which is called SAT, allows the analysis, not only of all the paths through the code but also all of the potential values of the variables in any given path. This level of precision means the latest static analysis tools can find very sophisticated defects at a tremendously low false positive rate (under 15%). And they can do all this over millions of lines of code in a matter of hours so developers get to spend their time implementing new features instead of tracking down those nasty bugs.

To see how far we’ve come with this challenge, maybe I can add some historical context. A number of your readers are probably familiar with Stephen Johnson, the creator of Lint, here’s what he had to say about false positives and software defects:

Lint tries to give information with a high degree of relevance. Messages of the form ‘xxx might be a bug’ are easy to generate, but are acceptable only in proportion to the fraction of real bugs they uncover. If this fraction of real bugs is too small, the messages lose their credibility and serve merely to clutter up the output, obscuring the more important messages.” Stephen Johnson, 7/26/1978.

So, as all your readers know, finding software defects is not a new problem – and clearly false positive results are not either. Of course, static analysis is not a silver bullet, and software development organizations still need to test their code and that will without a doubt uncover even more defects. But by leveraging static analysis the moment the code is written, thousands of developer hours will be saved by letting the computer start to do the debugging for them...

- Ben Chelf, Coverity CTO

Copyright © 2007 IDG Communications, Inc.

How to choose a low-code development platform