Use metrics to prove your IT project's worth

Is your IT project a dog or a delight? It all depends on how you measure it -- and who's doing the measuring

If you ever talk to Andrew Stewart, don't say the words "IT project."

As financial director at GAP Group, a $100 million equipment rental firm in Glasgow, Scotland, Stewart had to help his firm decide whether to spend $2 million on a new ERP (enterprise resource planning) system. Using Intentia's Opportunity Analyzer software, the company asked decision makers across the 700-person firm to define what they wanted the project to achieve and how to measure its success. There was only one rule: The phrase "IT project" was banned from all conversation.

"We didn't want our people thinking of it as something that has been forced on them by the technology department," Stewart explains. "For the system to work we needed buy-in from everyone."

Among other benefits, the opportunity analysis process resulted in a series of easily measured KPIs (key performance indicators). For example, building error-checking tools into the data entry system would increase accuracy from 95 percent to 99 percent, reducing billing mistakes. Automating administrative tasks would allow salespeople to make more calls each day, generating more revenue. By doing a better job of tracking idle equipment, the company could move heavy machinery closer to locations where it was needed, thereby getting a higher return on investment.

All told, GAP Group realized the new system could boost profits 10 percent per year -- a figure that would bring a smile to any financial director's lips.

This kind of careful project planning, however, is the exception, not the rule. More than half of all IT projects are over budget, off schedule, or fail to deliver the promised benefits, according to annual surveys by The Standish Group. That's because most companies don't do a good job of defining -- or measuring -- a project's goals.

Even worse, projects that come in on time, on budget, and to spec may still be perceived as failures. Although executives are often guilty of not clearly defining the business objectives -- or of continually moving the goalposts -- the IT side usually does an even worse job of communicating what it has accomplished. It's a double whammy that can spell doom for even the most promising projects. 

When bad things happen to good IT people

With some IT initiatives, success is easy to measure. If you're installing a CRM system that increases outbound sales calls or a knowledge-base app that leads to cuts in help desk staff, the ROI is clear. The bosses know what they're getting for their money, and everybody goes home happy, provided the project works as advertised.

But when it comes to services that keep the business humming, IT may be seen as little more than a drag on the bottom line -- and a lightning rod when things go wrong. Many tech managers find themselves in the awkward position of measuring their success by the bad things that haven't happened yet.

Click for larger view.

"Unfortunately, for many companies, IT is basically a utility," says Atwell Williams, director of enterprise service management at BMC Software. "When you come home and flip on the light switch, you don't immediately call the electric company and say, 'Thanks a lot guys, great job.' But if the lights don't come on, you better believe people will call."

So the Houston-based provider of system management solutions uses its own software, BMC Performance Manager, to monitor service outages that might have happened had they not been caught in time. By identifying and measuring the number of potential outages, BMC can establish goals for reducing them. Better still, BMC identifies the systems most important to the ongoing success of the enterprise and gives those priority when deciding what to fix first. 

"This is the essence of business service management," Williams says. "Many companies have lots of information telling them something bad is going to happen, but they don't know which of those things will truly impact their business. They end up spending time chasing down things that aren't really a problem."

At Sun Microsystems in Santa Clara, Calif., CIO Bill Vass sits in front of a custom dashboard monitoring the health of his "canaries" -- dummy users that log in to internal Sun systems across the globe every 15 minutes -- gauging response times, availability, and performance. When a system goes red, techs are automatically dispatched by pager to deal with the problem. In addition to obtaining real-time information, Vass and his team regularly comb detailed reports containing every conceivable metric, from code compliance to customer satisfaction -- the customers in this case being Sun employees.

"One of my rules is you manage what you measure," Vass says. 

But Sun has taken performance monitoring a step further by assigning a dollar value to every potential outage -- essentially marrying system performance directly to ROI. 

"At the end of every outage, we have a thing called 'estimated cost to Sun,' " Vass explains. "It took us eight months to determine the formula for that, and I had to get it blessed by finance before anyone would believe it. But now I can walk in and say, 'We need to upgrade network switches on this campus because if they fail, it will cost us $500,000 a month in downtime. New switches will cost $3 mil, but they'll pay for themselves in six months.' "

Perception is reality

One of the big challenges IT managers face is that of starting out in the hole -- with management and users anticipating the worst. Even if a project hits all its marks, it can be perceived as a failure.

"The problem with the word 'IT' is that it's almost always preceded by 'goddamned,' " says Ken Rau, senior consultant at Cutter Consortium in Wallace, N.C. "I've talked with CIOs who deliver 80 to 90 percent of their projects on budget and on time, but to their users it's, 'What have you done for me lately?' It's a communications and perception problem."

Vass agrees that bad tech experiences tend to linger longer than good ones. "One day on my morning commute there was an accident and it took me three hours to get to work. That happened five years ago, but I still remember it. People are like that with IT systems. They remember the one day in three years when the mail was down and they had to give a presentation. That's why we keep stats on things like system availability. So when they say, 'Your availability sucks,' we can show them the numbers that say we're up 99.999 percent of the time."

But many tech pros have only themselves to blame because they lack the necessary communications skills to promote themselves and their successes, says J. LeRoy Ward, executive vice president at ESI International, which provides training in project management and business analysis for the global Fortune 500.

Click for larger view.

"I don't think IT professionals do enough to sell themselves or their projects inside the organization," Ward says. "They tend to have very poor diplomatic skills and don't do a good enough job of relationship building."

Ward says a growing part of ESI's curriculum is teaching project managers how to act more like consultants, give good presentations, and hone their personal communication skills.

A key metric is how IT managers respond to a crisis and inform users. Both BMC and Sun measure how quickly tech managers communicate after a significant event and periodically poll user communities about their levels of satisfaction -- or lack thereof.

What managers say, and how they say it, is crucial to perception -- and harder to quantify.

BMC's Williams says the content of the message needs to be something business managers can relate to. "If you're the CTO for a bank, and you say something like, 'Router 63J5 just went down,' your users will have no clue whether to care about that," he says. "But if you say, 'Branch banking in the Northeast just went down,' you will definitely have their attention."

Susan Snedaker, author of How to Cheat at IT Project Management, says sometimes it's merely a matter of using the right spin.

"If you're having problems, don't say, 'We have no idea what the problem is or how to fix it.' Say, 'We're investigating the cause of the problem and will keep you updated.' "

Tech managers should look at every communication as another opportunity to create the impression of success, Snedaker says.

"One way to improve not only real but perceived project success is better communication with the project team, users, stakeholders, executives -- everyone who touches the project," Snedaker adds.

The CEO's projects never fail

The other key component of success is getting business owners to, well, own the project.

"If people think the project is owned by the IT department, as a kind of push from tech rather than pull from business, it dramatically decreases its chance of success," says Ulf Casten Carlberg, director of enterprise performance management at Intentia in Stockholm, Sweden. "Business management has to be in the driver's seat."

Bill Hill, IT director at the City of Dayton, Ohio, puts it a little more bluntly. "A project could be so good that it comes in well under budget, saves a fortune, and does everything it's supposed to do, but if it doesn't have a high-powered backer, it's a pig."

Conversely, Hill adds that a project could be a complete loser, but if someone uses it to get a favorable reaction from the press or public, it's considered a winner.

In addition to executive sponsorship, any major project needs to be closely aligned with bottom-line business goals. That means sitting down with all the departments that are affected and hammering out the project's objectives and KPIs, measuring the KPIs before the project to create a baseline, and conducting regular reviews to make sure the project is meeting its goals. It also means being willing to terminate failing projects with extreme prejudice. 

"The project needs to start with a good business case," ESI's Ward says. "You have to lay out why you're doing it, what you expect out of it, and have your business sponsor sign off on it -- literally put a piece of paper in front of him or her describing the project and ask them to sign it. You tend to look at something a lot harder when you're asked to put your signature on it."

And after a big project is under way, promote the hell out of it, says AMR's Shepherd -- give away hats and T-shirts to employees, talk about the project in the press and at user conferences, make sure it gets talked about in the annual report.

"When you do that, the organization becomes emotionally committed to the success of the project," Shepherd says. "It can be a major embarrassment or even career threatening if it fails."

At press time, GAP Group was nearing a final decision about which ERP system to adopt. Either way, Stewart was confident his firm would see good results from the move. Even though the project had barely begun, it was already well on its way toward success.

Copyright © 2005 IDG Communications, Inc.

How to choose a low-code development platform