Bad metrics still lead to bad management

If negotiating SLAs is good, is negotiating more of them better? Even if you accept the doubtful initial premise, the conclusion is seriously wrong

Dear Bob ...

We are beginning a Balanced Scorecard initiative here, and one of the proposed measurements is the "Number of SLAs." This doesn't seem like a good measure to me. This is supposed to measure internal customer satisfaction.

[ See also: "Why margin isn't a good metric" | Get sage IT career advice from Bob Lewis' Advice Line newsletter. ]

Seems to me we can keep adding new SLAs till the cows come home and say that we are doing great without accomplishing anything important.

I believe you have railed against this in past columns of Keep the Joint Running. Would you comment on this in Advice Line for me?


- Metric-ulated

Dear Metric'ed out ...

You just have to get me in for some metrics consulting. I'll even let you be the bad cop.

On the face of it at least, this appears to be a clear example of "let's measure what's easy to measure, not what's important to measure."

Layer on top of it the fallacy that IT considers the rest of the business to be its "internal customers" and you have a formula for serious mess-making.

Good metrics start with clear, well-defined, important goals. Not SMART goals: SMART stands for Specific, Measurable, Achievable, Relevant, and Time-bound, and makes measurability one of the criteria for goal selection -- a very bad idea -- without rating importance as being particularly important.

Good metrics start with important goals. Satisfying internal customers isn't an important goal because who cares if they're "satisfied," whatever that means. IT's job isn't to satisfy its internal customers. It's to help them succeed and in doing so help the company create and sell great products and deliver outstanding service to its real customers so that they'll come back and bring their friends.

Which makes measuring the satisfaction of internal customers an excellent example of Metrics Fallacy No. 2, which is measuring the wrong things. When you measure the wrong things, you get the wrong result no matter how well you measure them.

Let's pretend, though, that internal customer satisfaction is an important goal for some reason or other. That leaves open the question of whether the number of SLAs in place measures it appropriately (if not, it's an example of Metrics Fallacy No. 1, which is measuring the right things wrong). Since an SLA is a contract, it's equivalent to saying you can measure the level of external customer satisfaction by counting the number of customer contracts.

That doesn't work because contracts exist for when something goes wrong, not for when a relationship is smooth sailing. Even measuring the number of SLAs for which IT is in compliance doesn't measure satisfaction. It merely measures the number of internal agreements for which IT's performance is barely good enough.

Any organization that equates "just barely good enough" with "satisfactory" is setting the bar awfully low, in my opinion.

[ Looking for a full explanation of the four metrics fallacies and six characteristics of good measures? Get a copy of Bob's newest book, "Keep the Joint Running: A Manifesto for 21st Century Information Technology." You won't find a more sophisticated account of how to establish first-rate business metrics than its third chapter. ]

There are four metrics fallacies. The proposed metric hits two of them. I suspect the next two -- failing to measure important things and extending measures to individual employees -- are lurking just around the corner.

- Bob