Anatomy of an IT disaster: How the FBI blew it

What followed was the mother of all misunderstandings. In 2002, Hughes says, SAIC offered a proposal identifying December 2003 as the deployment date for VCF. He maintains, though, that as the changes rolled in, SAIC alerted the FBI that the cost and delivery date would be seriously affected. Ultimately, instead of a final version, what SAIC delivered in December 2003 was an incomplete system for evaluation.

Free IT resource Open Source Business Conference (OSBC) May 22-23, 2007 Sponsored by OSBC Free IT resource Virtualization Insights from Top Experts - Learn how virtualization gets real! Sponsored by Dell

The result, according to Hughes, was consternation. "Apparently the communication about what was going on in the project had not gone up the chain in the FBI and in [the Department of] Justice and in Congress;" he says, "and when we didn't actually deploy a completed system in December 2003, there was a lot of surprise at those levels."

Mueller voiced his bitter disappointment over the December version of VCF in the hearings. "When SAIC delivered the first product in December 2003, we immediately identified a number of deficiencies, 17 at the outset. That soon cascaded to 50 or more and ultimately to 400 problems with that software."

Hughes still bristles at having that prototype tested as a final version, but he does admit to some culpability. "That tells you one thing we should have done better: talking at every level in the government to make sure everybody was on the same page."

Downsized Delivery

In January 2004, the FBI hired a new acting CIO, Zalmai Azmi, its fifth in four years. Azmi officially took the CIO job in May 2004. An intense back-and-forth between Azmi and SAIC ensued during the first half of that year, with SAIC determined to nail down an unchanging set of requirements and Azmi pushing for delivery based on a contract that was performance-based, rather than cost-plus-award.

Both parties got their wish in June 2004, when Azmi and SAIC worked out a two-track plan. In December 2004, SAIC would deliver an IOC (initial operating capability), a workflow application that would automate the case-management document approval process. According to the OIG report, the FOC (full operating capability) was simply a new effort to "identify new requirements for developing a functional case management system to replace the ACS."

The IOC delivery is still a source of pride for Hughes. "That system contained 100 percent of the requirements that the customer said they wanted in the IOC system," he says.

The IOC system is currently being field-tested, but the FBI's take on it was somewhat different. At the hearings, Mueller and Azmi repeatedly alluded to a still-classified report by the nonprofit Aerospace Corporation that tallied up a long list of deficiencies in the IOC. Azmi also stated that the IOC represents only one-tenth of the VCF's intended capability, a claim that SAIC Executive Vice President Arnold Punaro rejected in interviews with reporters after the hearing, arguing that VCF never had a baseline to begin with.

Groundhog Day Redux

As SAIC labored on the IOC, still other plans were afoot. In September 2004, the FBI and the Department of Homeland Security began planning an interagency FICMS (Federal Investigative Case Management System) that would make the still-unfinished VCF obsolete.

The FICMS proposal sends FBI case management back to the drawing board at a time when, according to Hughes, SAIC's version of the VCF is further along than it may appear. "There are other capabilities that we have already designed and coded, and in fact the code is actually embedded in the IOC system -- but we turned it off and didn't test it or integrate it, because they didn't want it as part of the IOC."

Hughes estimates it will take three years for FICMS to become a working system. Meanwhile, he says, tests have already proven that SAIC's system, if deployed in stages, can scale to do what the FBI needs it to do. "I think it's crazy not to deploy it, regardless of how they want to go in the future," he says.

Today, however, everything remains uncertain. The pilot program for the IOC ends this month. The FBI says it will award a contract for the development of FICMS next month. If it does, then the FBI will likely abandon VCF.

After the fact, members of the Appropriations Committee have raised many objections to the way the VCF project proceeded: off-the-shelf software should have been considered, management should have been more forthright about problems, SAIC should have had its feet held to the fire.

The reality, though, is that IT backwaters like the FBI can't modernize without an extensible enterprise architecture, which the FBI admits it's only beginning to develop. Meanwhile, Gartner's Pescatore says only 76 percent of FBI personnel are using secure e-mail. Such outrages mean that, before any grand architecture rolls out, point solutions such as VCF must have their day.