Two of the most time-consuming aspects of debugging software are reproducing the problem and locating the cause of the bug. And as software becomes more complex -- with Web services integrating business processes and desktop environments moving to multithreaded software -- merely reproducing a bug can be a big project. If the developer cannot duplicate the event, though, resolving it is well-nigh impossible.

The Bottom Line AppSight 6.0 BMC/Identify, identify.com/ Very Good  8.6 criteria score weight Features 10 30% Interoperability 7 15% Performance 8 15% Management 9 10% Scalability 9 10% Ease-of-use 8 10% Value 7 10% Cost: Priced per console, starting at $150,000 for minimum configuration Platforms: Console: Windows, Java; data collection: Windows, J2EE servers Bottom Line: AppSight records very detailed data about program operations that enable developers to easily reproduce problems for quick bug identification. Version 6.0 is a superb tool that will save tech support, QA teams, and developers a lot of time -- if they can afford the breathtaking price and don’t need to run it on Linux or Unix servers. About our Reviews and Scoring Methodology

[ See also: InfoWorld Technology of the Year Awards Application Development winners ]

What developers need is a detailed log that records all events in a program’s execution, so a record exists when an anomaly occurs. Unfortunately, logging mechanisms often end up as home-grown, in-house systems based on tracing operations that typically record insufficient information and often change the behavior of the programs they monitor.

AppSight 6.0, from BMC/Identify (BMC bought Identify, the original developer of AppSight), is an extraordinary product that elegantly addresses the problems of reproducing and tracking down bugs. It is a tool that almost every developer will want to have, but one that I fear few will be able to afford.

Black boxes for all
AppSight uses logging software called Black Boxes that run on the systems to be monitored. These preconfigured installations can record almost every event on the host system, including network operations, file accesses, DLL loads and component accesses, mouse clicks and movements, video capture, and a slew of performance metrics.

Click for larger view.

On systems with accessible Microsoft .pdb files (generated at compilation), the Black Box maps events directly to individual lines of source code. A separate Black Box for J2EE systems performs similar logging; however, it drills down only to the class level.

Captured data is saved to a circular buffer -- that is, you pre-set the total amount of data you want to capture and the BlackBox will always capture exactly that much, overwriting older items when it runs out of room.

Beyond events, Black Boxes meticulously record system status. Virtually everything that is knowable about the hardware and software running at log time is captured by the Black Boxes, including version numbers of DLLs that can be crucial to isolating configuration issues.

Black Boxes are free -- you may use as many or as few as you wish. Preconfigured Black Boxes for various enterprise applications are available on the AppSight customer Web site. The boxes are coordinated by AppSight server software (included as part of the deployment), which acts as a repository for the logs and as a remote configuration tool for deployed boxes.

What you pay for are the consoles in which logged data is replayed. This is where all the magic occurs. And magic is the right word: When replaying a log, one console window shows exactly what the end-user saw, and the window below it displays the corresponding program events.

AppSight highlights code errors or anomalies in red. Click on the error (or any other event) and if you have captured detailed data, AppSight takes you to the active line of code. There, you can inspect the values of all variables in scope, including global variables, plus the call stack at the point the line was executed. And, as in a true debugger, you can step through the executed code, watching the variables change as you move along.

The interesting thing is that during this replay, you’re not actually running the software, you are simply stepping through the log. This provides several important advantages: you can go backwards through the events and replay them in order, or jump ahead without waiting for the program’s execution to catch up with you.

On Windows, logging is done from a separate process so that it interferes as little as possible with the running software (on the J2EE server, logging runs as an EAR file on the same JVM as the monitored application). AppSight does not require you to recompile an application. Instead, the Black Box uses code injection where needed to pick up the details.

All of this logging has a relatively small impact on your system performance. On most systems, Black Box performance overhead averages about 1 to 4 percent for most tasks, somewhat more if you turn on maximum capture -- which you would only do in the heat of a bug hunt.

The prototypical use of AppSight is for in-house QA and debugging. You put a Black Box on systems running a new application and watch it until you feel the software is sufficiently exercised. At that point, you might scale back the level of capture or remove the Black Box entirely. The big benefit of this approach is that developers and QA engineers no longer need to worry about not being able to reproduce a problem -- by playing the log, they can zoom in on the issue directly.

A site will generally use AppSight to monitor key systems as well as many client endpoints. If a problem occurs on an unmonitored desktop, you simply attach a Black Box to it and have the user repeat the process that caused the failure. Once you have the log, you remove the Black Box. The removal process truly removes all traces of the Box -- the code injection, the files, the logs, and any settings.

A few grumbles
Stepping through logs is an extraordinary sensation, like going backward in time. I could see much more of the code and the various interactions than in a standard debugging session. It’s a marvelous experience that would make any developer or QA engineer salivate.

There are some limitations. First, AppSight only works on Windows systems or J2EE servers -- there’s no support for Unix or Linux servers. Second, configuring Black Boxes is not a trivial task. BMC solves this in part by making a wide range of preconfigured boxes available. However, if you have to tweak one yourself, you’re better off calling BMC’s professional services.

Finally, there’s the breathtaking price. The $150,000 entry point for the smallest installation puts AppSight in a rarefied stratosphere that requires thoughtful ROI analysis. If you can live with the price, though, the product is a true gem.