In the early 1990s, Russian mobsters partnered with Italian Mafia families in Newark, N.J., to skim millions of dollars in federal and New Jersey state gasoline and diesel taxes. Special Agent Larry Depew set up an undercover sting operation under the direction of Robert J. Chiaradio, a supervisor at the Federal Bureau of Investigation's Washington, D.C., headquarters.

Depew collected reams of evidence from wiretaps, interviews, and financial transactions over the course of two and a half years. Unfortunately, the FBI couldn't provide him with a database program that would help organize the information, so Depew wrote one himself. He used it to trace relationships between telephone calls, meetings, surveillance, and interviews, but he could not import information from other investigations that might shed light on his own. So it wasn't until Depew mentioned the name of a suspect to a colleague that he obtained a briefcase that his friend had been holding since 1989.

"When I opened it up, it was a treasure trove of information about who's involved in the conspiracy, including the Gambino family, the Genovese family, and the Russian components. It listed percentages of who got what, when people were supposed to pay, the number of gallons. It became a central piece of evidence," Depew recalled during an interview at the FBI's New Jersey Regional Computer Forensic Laboratory, in Hamilton, where he is the director. "Had I not just picked up the phone and called that agent, I never would have gotten it."

A decade later, Depew's need to share information combined with his do-it-yourself database skills and connection to his old supervisor, Chiaradio, would land him a job managing his first IT project–the FBI's Virtual Case File.

Depew's appointment to the FBI's VCF team was an auspicious start to what would become the most highly publicized software failure in history. The VCF was supposed to automate the FBI's paper-based work environment, allow agents and intelligence analysts to share vital investigative information, and replace the obsolete Automated Case Support (ACS) system. Instead, the FBI claims, the VCF's contractor, Science Applications International Corp. (SAIC), in San Diego, delivered 700 000 lines of code so bug-ridden and functionally off target that this past April, the bureau had to scrap the US $170 million project, including $105 million worth of unusable code. However, various government and independent reports show that the FBI—lacking IT management and technical expertise—shares the blame for the project's failure.

In a devastating 81-page audit, released in 2005, Glenn A. Fine, the U.S. Department of Justice's inspector general, described eight factors that contributed to the VCF's failure. Among them: poorly defined and slowly evolving design requirements; overly ambitious schedules; and the lack of a plan to guide hardware purchases, network deployments, and software development for the bureau.

Fine concluded that four years after terrorists crashed jetliners into the World Trade Center and the Pentagon, the FBI, which had been criticized for not "connecting the dots" in time to prevent the attacks, still did not have the software necessary to connect any new dots that might come along. And won't for years to come.

"The archaic Automated Case Support system—which some agents have avoided using—is cumbersome, inefficient, and limited in its capabilities, and does not manage, link, research, analyze, and share information as effectively or timely as needed," Fine wrote. "[T]he continued delays in developing the VCF affect the FBI's ability to carry out its critical missions."

This past May, a month after it officially ended the VCF project, the FBI announced that it would buy off-the-shelf software at an undisclosed cost to be deployed in phases over the next four years. Until those systems are up and running, however, the FBI will rely on essentially the same combination of paper records and antiquated software that the failed VCF project was supposed to replace. The only recent addition has been a new "investigative data warehouse" that combines several of the FBI's crime and evidence databases into one. It was completed as the VCF started its final slide into oblivion. In addition, the FBI recently digitized millions of its paper documents and made them available to agents.

As the FBI gears up to spend hundreds of millions more on software over the next several years, questions persist as to how exactly the VCF went so terribly wrong and whether a debacle of even bigger proportions looms on the horizon. Despite high-profile Congressional hearings, hundreds of pages of reports churned out by oversight bodies, and countless anguished articles in the trade press and mainstream media, the inner workings of the project and the major players have remained largely invisible. Now, detailed interviews with people directly involved with the VCF paint a picture of an enterprise IT project that fell into the most basic traps of software development, from poor planning to bad communication.

Lost amid the recriminations was an early warning from one member of the development team that questioned the FBI's technical expertise, SAIC's management practices, and the competence of both organizations. Matthew Patton, a security expert working for SAIC, aired his objections to his supervisor in the fall of 2002. He then posted his concerns to a Web discussion board just before SAIC and the FBI agreed on a deeply flawed 800-page set of system requirements that doomed the project before a line of code was written. His reward: a visit from two FBI agents concerned that he had disclosed national security secrets on the Internet.