The IT worst case scenario survival guide

IT is a risky business. Here's how to avoid some common catastrophes and increase your chances of success

You’re lost in the IT wilderness, starved for funding and thirsting for recognition. As the infrastructure sinks slowly under your feet, alligators crawl out of their corner offices to snap at your heels and marketing weasels begin gnawing at your flesh ...

When you’re that far up the proverbial creek, and neither bailing out nor soldiering on seem like workable options, it’s easy to imagine that no one else in IT has ever been in such an impossible situation.

You’d be wrong about that. According to research by The Standish Group, one out of five IT projects fail outright, and more than half come in late or over budget. Why? The standard answer from the business side, “It’s IT’s fault,” conveniently ignores equally likely causes: bad requirements management, poor business planning, lousy communication, or the dreaded “scope creep.”

Here, we’ve identified five of the most common scenarios in which projects fail and what IT can do to avoid them. The stories you are about to read are true, although company names have been obscured to protect the guilty. Some date from the technology-at-any-cost ‘90s, while others are ripped from today’s headlines. But their lessons are timeless.

Scenario 1: Outsourcing run amok

Back in the mid-’90s, a large vending machine services firm wanted to save money by centralizing its operations, so it hired a Big Five consultant to implement a $20 million ERP system. Big mistake.

“Turns out, the consultant’s dream was to build a management information system from scratch because he thought he knew how to do it better than anybody else,” says Bob Price, former CEO of Control Data and author of The Eye for Innovation: Recognizing Possibilities and Managing the Creative Enterprise (Yale University Press, 2005). The result was a near total meltdown. Prices in the ERP system didn’t match those in the catalog, so customers refused to pay. The centralized system made it too expensive to collect from individual buyers, so revenues dropped. Midlevel managers who were never consulted about the system revolted and left in droves. The firm’s president lost his job over the fiasco, and his boss, the CEO of the firm’s parent company, ultimately resigned.

What do you do if the consultant you thought was a godsend turns into Godzilla?
1. Assess your internal talent. Nobody in the organization really understood the project, leaving the consultant totally in charge. If you don’t have the necessary expertise in house -- or are unable to hire it -- don’t take on the project. “It’s better to do business in a poor way,” Price says, “than undertake something you don’t understand and end up not doing business at all.”

2. Match the solution to your business. In this case, the firm was trying to force fit a centralized structure on an extremely decentralized business model. It ultimately solved the problem by putting sales and inventory management back out in the field but handling customer data centrally.

3. Avoid proprietary technology. The ERP system was not only a custom job, it was written in an obscure version of ALGOL (Algorithmic Language) -- a programming language the consultant loved but nobody else knew how to use, according to Price.

4. Manage it closely. Even the most minutely detailed consulting agreements can’t cover everything, especially as changes or clarifications are needed. Consultants can be very useful, Price says, but you need to state in no uncertain terms what you want them to do, and manage them very closely.

Scenario 2: The incredible expanding IT project
Three years ago, a major West Coast tech services company decided to roll out a Web-based content management system to handle its internal communications. But inexorably, the features list began to grow. Could they use the same system for customer support? Sure, said the systems integrator. How about selling research reports to clients? No problem. The budget for the project rapidly climbed to $100 million.

The catch? The systems integrator was also the software vendor who’d built the content management system. The vendor had never met a problem their software couldn’t solve -- for millions in additional development fees, naturally.

“By the time they contacted us, the company had spent closer to $280 million, and the percentage of test cases that actually worked was zero,” says George Kondrach, executive VP of Innodata Isogen, a content supply chain consultancy. Innodata recommended scaling down the project and bringing in third-party software to handle jobs the content management system wasn’t designed to do. Kondrach says Innodata could have fixed the problems for about $10 million, but that would have meant the client would have had to admit failure. Instead, the client continues to spend millions each year trying to make the system work.

How do you keep a project from spiraling into unfathomable depths?
1. Be vendor agnostic. Hiring an integrator with a vested interest or one whose expertise is limited to a single vendor’s products is a recipe for disaster, Kondrach says. Any proprietary solution can limit options later.

2. Know thy software. Understand the difference between a software feature and a new product category. Don’t try to use one system to do the work of five different applications.

3. Don’t bury your mistakes. Admitting failure early is one the best things you can do. “Don’t euphemize a project that’s failing as ‘making progress’ or ‘iterative development,’” says Kondrach. “It’s better to acknowledge that your initial design decisions are not workable.”

4. Know when to give up. “Don’t throw good money after bad,” says Patrick Gray, president of Prevoyance Group, a New York-based consulting firm. “It’s painful, but it’s still better to waste $100 million than $200 million.”

Scenario 3: The hacker from hell
Last year, the Web site of a large financial services firm was hacked. Acting quickly, the company took down its primary site and brought a duplicate one online with virtually no service hiccups. But what the firm didn’t know was that the attacker had planted malicious code on the site months earlier -- enough time for his backdoor to be propagated into all the site’s backup sets. So when the second site came online, the hacker continued accessing user accounts. The firm had to go completely offline for several hours while it identified and fixed the problem.

By relying on a single application or platform for core business functions, you create a single point of failure, says John Pironti, principal security consultant for Unisys. In this case, the same flawed code was used in the firm’s primary and backup Web sites -- which the hacker happily exploited.

How do you keep hackers from finding your Achilles’ heel?
1. Diversify platforms. Enterprises need to put their core business functions on multiple platforms and keep data synchronized among them. So, for example, if the Windows platform is compromised, the organization can fire up the Linux implementation and operate with minimal interruption.

2. Check your backups. Organizations should verify their code before they back it up or restore it, says Pironti. The easiest way is to create a hash from compiled code before it’s put into production, then do the same thing before each backup. If the hashes aren’t identical, the code has been tampered with -- and the organization will discover it within hours, not months.

3. Audit processes. Most organizations need to do a better job of logging system activities and correlating them into events. By establishing a range of what “normal” transactions look like, organizations can more easily detect and respond to anomalous behavior.

4. Plan for small disasters. Most enterprises have their own worst-case-scenario guidebook on how to handle huge disasters but don’t have a clue what to do if just part of their system -- such as a Web site -- breaks down. They need a plan for every part of the puzzle, says Pironti.

5. Think business, not IT. Organizations should approach security and continuity with the idea of doing whatever it takes to keep the business going -- even if that means reverting to pen and paper, says Pironti. “If institutions looked at things from the perspective of business processes and not technology, they would develop much better vulnerability management plans,” he says.

Scenario 4: Software of the damned
Back in the late ‘90s, a large, Midwest maker of consumer products thought a new ERP system would be just the ticket. All the top executives signed off on it, and the IT department got busy. Many months and $40 million later, the project was finally done -- and users wouldn’t go near it.

“When they finally flicked the switch, they had a near-total rebellion from their users,” says Phil Bloodworth, U.S. leader of the IT Effectiveness practice at PricewaterhouseCoopers (PwC). Nobody had bothered to talk to the people who were supposed to use the thing. Some modules were retrofitted for use in other departments, but the bulk of the system was abandoned.

How do you keep your expensive project from becoming a lifeless zombie?
1. Talk to users. The bigger the project, the more important communication becomes, says Joel Koppelman, CEO for Primavera Systems, a maker of project management software. “The project itself may be done perfectly, but if the people are not prepared for the change, it will fail.”

2. “Socialize” the project. Have users help spec out the project and evaluate candidates, advises Bloodworth. “Get everyone to buy in by asking them what their needs are, what the current system does well and what it does poorly, and what’s required for the job.”

3. Avoid “Scope Creep.” Keep the feature list in check by putting a price tag on every change request. “A user or manager’s request for a ‘critical’ change may become less critical when it is determined the associated change would burn $1.7 million,” Gray says.

4. Keep training. “You need ongoing training, both before and after you convert to the new system,” Bloodworth says. “Tell users, ‘That [old] report is now this new report; your new screen will look like this.’ “

5. Break it down. “Multi-year, monolithic IT projects are a sure-fire recipe for failure,” says Michael Hugos, a former CIO of an $8 billion distribution cooperative and author of Building the Real-Time Enterprise: An Executive Briefing (John Wiley & Sons, 2004). “The key to success is breaking big projects into smaller, self-contained pieces that can be put into production within three to nine months.” The business side can see what it’s getting for its money -- and bail on the project if it fails to meet expectations.

Scenario 5: The IT department tied in knots
PwC’s Bloodworth says he was once called in to consult with a large, Midwest entertainment company where technology was viewed as a necessary evil, unconnected to the company’s core business. Even seemingly simple changes required huge lead times. The backlog on pending projects was enormous. Nothing ever got done, and IT was viewed as a dismal failure.

But Bloodworth discovered the fault lay not with the tech department but with how the company managed -- or rather, mismanaged -- projects. There was no process for approving projects, prioritizing them, or determining which projects were too old or marginal to pursue. “We found a portfolio hundreds of projects deep,” he says. “IT just focused its resources on whatever group shouted the loudest.”

How do you cut through the tangle of demands and make IT functional again?
1. Speak truth to power. “You need to have someone in the organization with the ability to take a step back and say, ‘This is out of control; we have a problem we need to fix,’” says Bloodworth. “If no one’s willing to step up, you may need to bring in an outside consultant who can tell the CEO, ‘We have a governance issue, and it starts with you.’”

2. Limit the number of decision makers. One of the first things PwC did was build an executive committee to approve IT projects, along with a second-tier steering committee for day-to-day management -- and made everyone accountable for cost. “Putting in a formal process really helped them,” Bloodworth says.

3. Analyze the portfolio. Scrap anything that’s old or not aligned with core business goals, then prioritize what’s left.

4. Elevate IT’s stature. IT has to be about more than just keeping all the systems glued together every night, says Scott Christian, a director in PwC’s IT Effectiveness practice. To create an environment where tech is viewed as vital to success, organizations must empower techies to be strategic thinkers and not just order takers. They may also need to parachute in an enlightened CIO who’s a businessperson first and a techie second.

Join the discussion
Be the first to comment on this article. Our Commenting Policies