How software cut the world's fifth-largest company in half

This is a story of what happens when overconfidence gets in the way of safe and sound software development practices

IT
IDGNS

Long ago in a company far, far away, I watched a 10-digit disaster project cut the world's fifth largest company in half.

Actually, it wasn't a single defect, it was an architectural fiasco. It's not important which company it was -- just call it BTF (Bet the Farm) --  but how it happened is a lesson still unlearned by too many companies until they have their own 9 (or more)-digit defect.

Although a conglomerate, BTF enjoyed a dominant global position in its primary product line. Many of BTF's companies were competing for investment, but senior management committed the bulk of investment capital to renovate its primary product line responsible for roughly half BTF's revenue. BTF intended to reinvent an analog product using digital technology programmed in a new language. At the same time BTF consolidated all its previously independent, international development centers into one large integrated project.

BTF envisioned a generic digital architecture that could be delivered globally with minor modifications to satisfy local regulations. It designed specialized microprocessors, each of which supported specialized software that collectively executed the product's functionality. The company hired the best and brightest from competitors who had already developed similar products, but not on such a grand scale with such challenging requirements.

This brain trust spent several years designing and implementing the new system across the spread of development centers. Three months before the deadline for delivering the first system to a competition for the business of Country A, senior management wanted to know if BTF had a system to deliver. Apparently, they had not been receiving regular management reports from the various development centers with accurate data on progress.

The request for accurate progress data bounced down 7 layers of management until it was assigned to the software measurement guy -- me. I flew to the site coordinating all development work and promptly began looking for progress data, which was not to be found. I decided to look at the integration test reports to see how the system was coming together. As I charted the progress of components through integration testing, I noticed that over the last few months, the number of components entering test had increased exponentially. They were shoveling software into test as fast as they could code it.

Then came the weird part. Component IDs went into integration test and never came out. Component IDs were coming out of integration test that had never gone in. In fact, there were twice as many component IDs coming out as had had gone in! What?!?

The requirements kept growing without limit, even though most countries only required a basic version of the product. The software had grown larger than the memories on the chips (remember this was long, long ago). When it was no longer possible to deny the misfit between load modules and their assigned chips, the architects redesigned the entire system during the middle of integration test. They dismantled the elegant software architecture and dispersed components across myriad chips. The generic architecture was dead.

Facing the impending competition deadline, Country A's development center executive told the other development centers to disappear. They would modify the system to meet Country A's basic requirements and forego the sophisticated, but excess functionality. In a trash can, I found a monthly management report showing that progress toward delivery was linear and insufficient to meet the deadline. These monthlies were never sent to corporate headquarters. Progress was limited by an insufficient number of people in Country A who understood the new programming language and could fix bugs. Knowledgeable developers by the planeload were immediately shipped to Country A. Through brute force they delivered on-time and won Country A's competition. Everyone celebrated!

Shortly thereafter, BTF won competitions in Countries B, C, D, etc. Then it came flying apart. With the generic architecture in shreds, the cost of modifying the system to meet each new country's regulations exceeded the profit. When BTF couldn't complete the required modifications in time to win their largest national market, the game was over. The fifth largest company in the world sold off a product line comprising roughly half the company's business for less than the product line's annual revenue.

The obvious lesson is don't let product capabilities grossly exceed the market's actual requirements. But there are other lessons here, primary among which is limiting sources of risk when engaging new technologies. BTF absorbed three titanic sources of risk in the same project:

  1. A big bang transition from analog to digital technology for the whole product line
  2. A new development language with a limited base of experienced developers
  3. A consolidation of development centers that had each treasured their national independence

It was far too much change driving far too much risk. Risk that mangled a system architecture into a 10-digit defect. Risk that senior management was not tracking. Risk that resulted in a working product cutting the world's fifth largest company in half.

This article is published as part of the IDG Contributor Network. Want to Join?