Measurement: a two-edged sword

Too of­ten, the prac­tice of mea­sur­ing things is used in place of ba­sic think­ing, and, log­i­cal­ly, leads to low cus­tomer sat­is­fac­tion and poor prod­uct qual­i­ty.

The ma­jor rea­son for that is that in most cas­es, mea­sure­ment is de­hu­man­iz­ing. A cus­tomer, that is a real, ac­tu­al per­son, be­comes a num­ber, or, worse, just a part of some sta­tis­ti­cal re­sults. In­stead of hu­man re­la­tions, we have tick­ets, and num­bers, and charts, and work­flows. And we care more and more about our per­for­mance in­dex, or the num­ber of cus­tomers we served, or the av­er­age num­ber of stars cus­tomers gave us, rather than about per­sons them­selves.

To­day, I re­ceived a mes­sage from a sup­port guy from some com­pa­ny. I opened a tick­et a week ago, and the prob­lem I was de­scrib­ing ap­peared more com­pli­cat­ed than I thought at first. To­day's mes­sage in­clud­ed a sen­tence which is an ex­cel­lent ex­am­ple of met­rics vs. per­sons dis­cus­sion:

[...] We can not keep the tick­ets open for these many days as it vi­o­lates our SLA. [...] Will be mark­ing these tick­et as close.

Clear­ly, re­spond­ing to cus­tomers isn't their goal. Their goal is to close as many tick­ets as fast as pos­si­ble. At the root, the prob­lem is sim­ply a bad­ly cho­sen mea­sure­ment.

SLA makes per­fect sense for an ISP or an en­er­gy com­pa­ny. If I have no elec­tric­i­ty in my house for one hour per day, there is a slight risk that I'll be tiny lit­tle up­set, and might switch to a com­peti­tor who can ac­tu­al­ly de­liv­er near­ly un­in­ter­rupt­ed pow­er. While psy­cho­log­i­cal im­pli­ca­tions of the per­cent­age of down­time are non-lin­ear and not ob­vi­ous to mea­sure, it is still fair to mea­sure the down­time per­cent­age and to have a lin­ear or hy­per­bol­ic re­la­tion with the mon­ey the com­pa­ny should re­fund me.

What makes it easy to have such math­e­mat­i­cal re­la­tion be­tween the met­ric and the ac­tu­al fi­nan­cial con­se­quences is the cor­re­la­tion be­tween the SLA and the con­se­quences in terms of per­ceived qual­i­ty of ser­vice. An in­ter­rup­tion in pow­er sup­ply, in­de­pen­dent­ly of the time of day when it hap­pens or the cus­tomer it af­fects, will al­ways be per­ceived neg­a­tive­ly. Turned in a dif­fer­ent way, there is no rea­son­able jus­ti­fi­ca­tion to in­ter­rupt the ser­vice, which would lead to in­creased cus­tomer sat­is­fac­tion.

When ap­plied to the time sup­port takes to an­swer the ques­tions, both el­e­ments stop mak­ing any sense:

• The fact that a com­pa­ny took two days to an­swer my ques­tion doesn't mean I'll be twice less sat­is­fied than if it took one day. I might get an­gry that it takes so long if the ques­tion is ur­gent and easy to an­swer. Or maybe I wouldn't care if the ques­tion is not ur­gent.

• More im­por­tant­ly, oth­er fac­tors are much more im­por­tant than the time-to-an­swer de­lay. The most cru­cial as­pect is the qual­i­ty of the an­swer it­self. I'd rather pre­fer an in-depth an­swer from some­one who took time un­der­stand­ing the prob­lem and ac­tu­al­ly try­ing to solve it, even if I have to wait for it, than a quick “RTFM” thrown at me in a con­text where I clear­ly ex­plained that the prob­lem I'm en­coun­ter­ing is not doc­u­ment­ed.

Ob­sess­ing over met­rics—and this is what hap­pens when the wage, or the po­ten­tial pro­mo­tion, or even the rep­u­ta­tion of a giv­en per­son in a com­pa­ny is based on pure sta­tis­tics—makes things even worse. It makes us for­get what are the ac­tu­al things which mat­ter, and fo­cus on things which are mea­sured. This leads to the an­swers such as the one I re­ceived.