How the FBI's upgrade spelled disaster

For one crime-fighting agency, good intentions weren’t enough to prevent failure

This week’s issue brings an insightful article by Executive Editor At Large Eric Knorr on a dream IT project that devolved into a nightmare.

In an odd twist, the victim — or perpetrator, depending on your perspective — of this melodrama was the FBI, which normally is in the business of helping victims by snagging perpetrators, not playing either role.

As Knorr describes it (see “Anatomy of an IT disaster: How the FBI blew it”), the bureau set out in the late 1990s to update an outmoded infrastructure — think mainframes and 386-based PCs. The project grew to include the perfectly sensible goal of digitizing key portions of its paper-based case tracking system.

Just about everything went wrong. Knorr provides the gory details, but here are a few observations.

Some projects are doomed to fail, and the FBI’s so-called Trilogy project had several earmarks. For one, requirements never stopped changing. Plus, many revisions were driven by events beyond anyone’s control — most notably the Sept. 11 terrorist attacks, which led to a thorough shake-up of government security agencies. In that environment, Solomon himself could not have defined a stable set of needs.

Similarly, companies often fall in such traps when they undertake a bet-the-farm IT makeover in the face of dramatically changing market conditions. Ironically, this is exactly when most companies try something dramatic, as obvious dangers mitigate management’s normal aversion to risk. Such projects have a chance of succeeding, I suppose, if the market doesn’t suddenly change again and if management’s plan is sound. Sadly, neither is guaranteed — and when outside forces perpetually reshape internal priorities, the outcome is often unhappy.

Large projects should proceed in small steps, each of which should be an improvement even if the rest of the plan never happens. That’s hard to do in the real world, of course, and Trilogy suffered an additional twist — a little gotcha called the “flash cutover.” The idea was to replace the old system at a single stroke. There are industries in which such draconian measures are necessary — and crime fighting may be one of them — but Murphy’s Law guarantees the transition will be painful.

Change must start at the bottom. Much of the Trilogy project remains secret. But apparently, the original plan would have done little to reform an antiquated paper-based case management system in which, according to analysis from the Department of Justice, information could not be changed or updated electronically. That was probably a mistake.

Hindsight is 20/20, surely, and Knorr suggests there is plenty of blame to go around. Fortunately, there are also valuable lessons.