In a blog post, James Bach discusses the virtues of dumping Key Performance Indicators (KPI’s). It’s a shock to think that maybe we should get rid of quantitative metrics, however, there is a sound argument behind the idea.
If you think about it carefully enough, quantitative metrics serve two purposes:
- Snapshots
- Objectifying reality
Snapshots
As an experienced software performance optimizer, I can attest to the value of snapshots. Without objective metrics there is no way to narrow down what is consuming time in a software system. However, there are a couple of important points to remember about capturing metrics in software development — and by extrapolation the real world.
- Capturing good metrics adds a lot of overhead, especially to repetitive operations.
- If you’re not adding a lot of overhead, you’re not capturing good metrics.
- Once you’ve optimized the processes, you need to get rid of the overhead that is capturing the metrics.
Objectifying reality
This is a fancy term for vanity metrics. The problem with metrics is two fold: you can only manage what you measure and when you measure, that’s what you get — whether or not you wanted it. Usually, you discover that what you are measuring isn’t what you really want. That leads to
In Bach’s words
Gather relevant evidence through testing and other means. Then discuss that evidence.
That’s how it works for us. That’s how it works for publishers. That’s how it works for almost everything.
In conclusion
I’m not against KPI’s and metrics in and of themselves. I’m opposed to blindly relying upon metrics to replace qualitative discussions. Numbers are a hollow model of reality and are too easy to use to justify distortions. Until you really understand what the number means, it’s a mistake to put too much faith into it.