This isn't an isolated case. SMART isn't always very smart because as a general rule, the more important the goal, the harder it is to establish metrics for which data collection is affordable and the data itself is objective.
Anyone who has worked in IT knows this, because almost no matter what you're responsible for implementing, the cost is easy to measure. However, minor matters such as sound engineering, maintainable interfaces, and extensibility for future needs are nearly impossible to objectively gauge.
The four fallacies of metrics
There are, it turns out, four different ways to do metrics wrong. You can:
- Measure the right things badly.
- Measure the wrong things, either well or badly.
- Neglect to measure something important.
- Extend measures to individual employees.
The first problem is the easiest to avoid. Once you know what you need to measure -- what your goals are -- the most common glitches are easy to spot and remedy. A common example is failing to weight different kinds of cases differently. Our help desk example would have failed this test, even if resolution rate was the right thing to measure: All calls to the help desk were counted equally, even though different kinds of calls would be expected to take dramatically different amounts of time to fix.
The second problem is harder to spot. It was the second metrics sin committed by the CIO in question. Resolution rate wasn't the right element to measure. How much user time at work is free of technical difficulties is what matters, whether it's the result of prevention, rapid resolution, or even dumb luck.
The CIO continued his unbroken streak with metrics fallacy No. 3. Companies should want their employees to take maximum advantage of the tools available to them. It's another very important and hard-to-measure goal. The help desk manager recognized its importance and instituted programs to move in that direction. The CIO, by failing to measure it, ensured the death of those programs. Anything you don't measure you don't get.
That leaves the fourth and most controversial metrics fallacy -- extending metrics to individual employees. Tempting as it is, it's almost always a losing proposition, because smart (as opposed to SMART) employees will almost always figure out ways to game whatever metrics you apply to them.
If you managed the help desk in question or worked on it as an analyst, would you resist the temptation to ask every friend you had in the business to call in on a regular basis with easy-to-fix problems? Maybe you would. Then again, what if your raise or bonus depended on the metric and youd already lost respect for the powers that be because they were working so hard to keep you from doing what mattered most? I'm guessing that if you resisted the temptation, not only would you be the exception, but you'd be the exception most likely to be included in the next round of layoffs.
It's the right course of action, isn't it -- laying off the company's poorest performers?
Metrics do matter. Without some way of knowing whether or not the organization is achieving its most important goals, its managers are flying without instruments. The challenge is getting them right because there are worse things than flying without instruments. Flying with instruments that provide false readings is, for example, much, much worse.
This story, "The four fallacies of IT metrics," was originally published at InfoWorld.com. Read more of Bob Lewis's Advice Line blog on InfoWorld.com. For the latest business technology news, follow InfoWorld.com on Twitter.