Home Home Posts Rants about IT DevOps Stuff I'm working on

Measurement: a two-edged sword

Arseni Mourzenko
Founder and lead developer, specializing in developer productivity and code quality
September 18, 2015

Too often, the practice of measuring things is used in place of basic thinking, and, logically, leads to low customer satisfaction and poor product quality.

The major reason for that is that in most cases, measurement is dehumanizing. A customer, that is a real, actual person, becomes a number, or, worse, just a part of some statistical results. Instead of human relations, we have tickets, and numbers, and charts, and workflows. And we care more and more about our performance index, or the number of customers we served, or the average number of stars customers gave us, rather than about persons themselves.

Today, I received a message from a support guy from some company. I opened a ticket a week ago, and the problem I was describing appeared more complicated than I thought at first. Today's message included a sentence which is an excellent example of metrics vs. persons discussion:

[...] We can not keep the tickets open for these many days as it violates our SLA. [...] Will be marking these ticket as close.

Clearly, responding to customers isn't their goal. Their goal is to close as many tickets as fast as possible. At the root, the problem is simply a badly chosen measurement.

SLA makes perfect sense for an ISP or an energy company. If I have no electricity in my house for one hour per day, there is a slight risk that I'll be tiny little upset, and might switch to a competitor who can actually deliver nearly uninterrupted power. While psychological implications of the percentage of downtime are non-linear and not obvious to measure, it is still fair to measure the downtime percentage and to have a linear or hyperbolic relation with the money the company should refund me.

What makes it easy to have such mathematical relation between the metric and the actual financial consequences is the correlation between the SLA and the consequences in terms of perceived quality of service. An interruption in power supply, independently of the time of day when it happens or the customer it affects, will always be perceived negatively. Turned in a different way, there is no reasonable justification to interrupt the service, which would lead to increased customer satisfaction.

When applied to the time support takes to answer the questions, both elements stop making any sense:

  • The fact that a company took two days to answer my question doesn't mean I'll be twice less satisfied than if it took one day. I might get angry that it takes so long if the question is urgent and easy to answer. Or maybe I wouldn't care if the question is not urgent.

  • More importantly, other factors are much more important than the time-to-answer delay. The most crucial aspect is the quality of the answer itself. I'd rather prefer an in-depth answer from someone who took time understanding the problem and actually trying to solve it, even if I have to wait for it, than a quick “RTFM” thrown at me in a context where I clearly explained that the problem I'm encountering is not documented.

Obsessing over metrics—and this is what happens when the wage, or the potential promotion, or even the reputation of a given person in a company is based on pure statistics—makes things even worse. It makes us forget what are the actual things which matter, and focus on things which are measured. This leads to the answers such as the one I received.