Anatomy of an IT disaster: How the FBI blew it

Some FBI agents ruefully refer to the trilogy project, a massive initiative to modernize the FBI's aging technology infrastructure, as the "Tragedy" project. It certainly has all the earmarks of tragedy: the best intentions, catastrophic miscommunication, staggering waste.

Free IT resource Open Source Business Conference (OSBC) May 22-23, 2007 Sponsored by OSBC Free IT resource Virtualization Insights from Top Experts - Learn how virtualization gets real! Sponsored by Dell

Trilogy, as the name suggests, had three parts: an enterprisewide upgrade of desktop hardware and software; deployment of a modern network infrastructure; and an integrated suite of software for entering, finding, sharing, and analyzing case information. In a congressional hearing last month, FBI Director Robert Mueller was careful to note that the first two parts of Trilogy have been completed: no less than 30,000 computers, 4,000 printers, 1,600 scanners, 465 servers, and 1,400 routers were deployed as of April 2004.

After more than four years of hard work and half a billion dollars spent, however, Trilogy has had little impact on the FBI's antiquated case-management system, which today remains a morass of mainframe green screens and vast stores of paper records. As Senator Judd Gregg observed, "the software, which runs the hardware, is a huge problem."

The problem with that software, known as VCF (Virtual Case File), is that it isn't in production and may never be. VCF may be one of the most extreme examples of requirements bloat in IT history. What began as a fairly modest software project swelled into an all-encompassing replacement for a panoply of woefully outmoded applications and procedures. Along the way, the FBI went through five different CIOs, 10 project managers, and 36 contract changes. The result, said Senator Patrick Leahy at February's Senate Appropriations Committee hearings, "has been a kind of train wreck in slow motion."

Accounting for $170 million of Trilogy's $581 million price tag, VCF fell afoul of extraordinary circumstances --notably, the Sept. 11 attacks, which piled enormous pressure onto the Trilogy project and altered the course of VCF dramatically.

FBI representatives declined to be interviewed for this story. But thanks to the Senate hearings, a report from the Department of Justice's Office of the Inspector General (OIG), and interviews with the FBI contractor that developed VCF, InfoWorld has been given a rare glimpse into the inner workings of a colossal IT failure.

Green Screens and Filing Cabinets

Sen. Leahy offered another, more whimsical analogy for Trilogy: the 1993 movie Groundhog Day, in which Bill Murray wakes up each morning to relive the same day. Since 1997, proposals for modernizing the FBI's technology and processes have emerged again and again, culminating with Trilogy. Trilogy itself then underwent a cyclic series of evaluations and funding requests until Congress finally learned that its third leg, VCF, might never materialize.

For the foreseeable future, that leaves the FBI with its obsolete, mainframe-based ACS (Automated Case Support) system, which requires the user to traverse a dozen 3270 green screens to upload a single document. Worse, according to the OIG's report, "the ACS only serves as a backup to the FBI's paper file system [and] information within that system cannot be changed or updated."

By the year 2000, aging infrastructure alone -- including 386-based desktop PCs and 12-year-old interoffice networks -- was hobbling the FBI. In September 2000, the FBI proposed FITUP (FBI Information Technology Upgrade Project), for which Congress allocated $379.8 million, spread over an estimated three-year effort. Two months later the project was divided into three parts and renamed Trilogy.

Trilogy's requirements for desktop hardware and network infrastructure stayed relatively stable throughout the life of the project (although an additional $100 million was allocated to accelerate completion, to no avail). The software portion was a different story.

What we now know as VCF was preceded by Trilogy's original and quite different proposal, UAC (User Applications Component). The objective of UAC was simple: "Webify" five of the 42 mainframe applications employed by FBI agents in the course of investigations. On the face of it, this seems like a sensible first step toward modernization, but it can also be seen as the crucial error of the entire Trilogy project.