Measuring quality like a pro

My pre­vi­ous ar­ti­cle led some of the read­ers won­der­ing how a cor­rect ap­proach to mea­sur­ing stuff looks. I'll use this as an op­por­tu­ni­ty to show how I worked my­self as a con­sul­tant. The ar­ti­cle would re­trace my con­sult­ing ex­pe­ri­ence from 2016 for one small French com­pa­ny (about thir­ty per­sons) de­vel­op­ing a soft­ware prod­uct used in a very nar­row field with very few cus­tomers. The com­pa­ny shared this mar­ket with a com­peti­tor of rough­ly the same ex­per­tise lev­el. The com­peti­tor ap­peared in this mar­ket three years ago, and over time “stole” an im­por­tant num­ber of cus­tomers.

The founder of this com­pa­ny came to me with a ma­jor con­cern: there were too many bugs and re­gres­sions in the soft­ware prod­uct. While every­one un­der­stands, in­tu­itive­ly, that bugs and re­gres­sions are bad, the sole pres­ence of bugs and re­gres­sions is not a use­ful met­ric. One com­pa­ny may have lots of them, while re­main­ing suc­cess­ful; an­oth­er one could have few bugs, which would se­vere­ly im­pact the busi­ness. The goal of the first meet­ing was to qual­i­fy what the boss ac­tu­al­ly meant by “too many bugs and re­gres­sions.” The meet­ing re­vealed that there are four is­sues caused by the bugs and re­gres­sions in pro­duc­tion, which have a di­rect fi­nan­cial im­pact:

• When a cus­tomer finds a bug, he calls the sup­port. The sup­port wastes time try­ing to find what's wrong, then dis­cov­ers that the prob­lem is like­ly to be re­lat­ed to what de­vel­op­ers did. The sub­ject is es­ca­lat­ed to the de­vel­op­ers. The de­vel­op­ers waste their time solv­ing the bug, and then an hot­fix has to be de­ployed to pro­duc­tion. The de­ploy­ment step takes one day of work. Even a tiny lit­tle bug would take at least two man days to be solved.
• The im­age of the com­pa­ny suf­fers. For in­stance, a month be­fore our meet­ing, an im­por­tant cus­tomer moved to the com­peti­tor, be­cause of the con­stant is­sues with the prod­uct. More­over, since there are about three hun­dred po­ten­tial cus­tomers in France in this sec­tor, they do talk to each oth­er oc­ca­sion­al­ly, and some may pre­fer the prod­uct of the com­peti­tor if they hear some­one else talk­ing about all the bugs he had found in the prod­uct of my client.
• Be­cause of high stress, de­vel­op­ers are of­ten sick and oc­ca­sion­al­ly have burnouts.
• The time which is spent by the de­vel­op­ers fix­ing bugs is not spent on some­thing ac­tu­al­ly use­ful, that is, the new fea­tures and en­hance­ments. One con­crete prob­lem is that on sev­er­al oc­ca­sions, the com­peti­tor re­leased the same fea­tures months ear­li­er, at­tract­ing more cus­tomers who need­ed those spe­cif­ic fea­tures.

The fact that it takes one day to de­ploy to pro­duc­tion sur­prised me, so we went to see the team man­ag­er for a ses­sion of five whys:

• Why does it take one day to do a de­ploy­ment? Be­cause the de­ploy­ment pro­ce­dure is man­u­al.
• Why is the pro­ce­dure man­u­al? Be­cause it is too com­plex to au­to­mate.
• Why is it too com­plex? Be­cause there were too many changes over time, and no­body has the skills to refac­tor the process.
• Why is no­body has the skills? Be­cause the orig­i­nal guy who cre­at­ed the thing left the com­pa­ny, and no one un­der­stands the idio­syn­crat­ic ap­proach­es which were used.
• Why are the ap­proach­es con­sid­ered idio­syn­crat­ic? Be­cause the guy didn't know any stan­dards back then, and used an in-house ap­proach, rein­vent­ing the wheel.

Not the most con­struc­tive five whys, but it seems that the num­ber of bugs in pro­duc­tion is not the only prob­lem in this com­pa­ny. Spend­ing one day do­ing a er­ror-prone pro­ce­dure by hand is not par­tic­u­lar­ly en­joy­able. There is a very con­crete risk of mak­ing er­rors, and er­rors were made reg­u­lar­ly. When I asked how many de­ploy­ments the team were do­ing per year, the man­ag­er told that it should be about six­ty. That's three man-months, large enough to be able to hire an ex­pert for sev­er­al months to au­to­mate the thing.

Soon, we start­ed work­ing on a se­ries of met­rics:

• We can track over time how much time the sup­port spends work­ing on the bugs and re­gres­sions. Peo­ple from sup­port had an habit, un­like the de­vel­op­ers, to work with the tick­ets and track what­ev­er they do, so such a met­ric was easy to col­lect. The idea was that it would give us a very con­crete met­ric of the time wast­ed, and we could com­pare how it evolves over time. It was re­al­ly about the du­ra­tion, and not the num­ber of bugs: some bugs are rel­a­tive­ly easy to spot, and it takes a few sec­onds for the sup­port to know that they need to es­ca­late it to the de­vel­op­ers, while some oth­ers are cryp­tic enough to waste a few hours.

  The met­ric can be gamed with de­vel­op­ers putting bet­ter logs, for in­stance. I'm per­fect­ly hap­py with that, since it still achieves the tar­get of less time be­ing spent on bugs.

  In ret­ro­spec­tive, this met­ric was a bad idea, and it wasn't com­mu­ni­cat­ed cor­rect­ly. The folks from sup­port mis­un­der­stood its pur­pose. At the be­gin­ning, they thought that the man­age­ment will look at how much time the team mem­bers were spend­ing, in or­der to know who works more. The man­age­ment ex­plained that this is not the pur­pose, and that the met­ric would cov­er the whole team, and not in­di­vid­ual mem­bers. This cre­at­ed an­oth­er is­sue: sup­port guys were afraid that the man­age­ment would still start ask­ing why the whole team spends so lit­tle time do­ing the work they are paid for, an so they start­ed to ar­ti­fi­cial­ly in­crease the time. The dif­fer­ence is clear­ly vis­i­ble in Fig. 1, which shows the time spent by the sup­port on the tick­ets re­lat­ed to bugs and re­gres­sions in pro­duc­tion. Ex­cept the mo­ment where 1 every­body goes on va­ca­tion, and the mo­ment 2 that the sup­port guys will re­mem­ber the rest of their lives as “the Black Week,” one can see some­thing hap­pen­ing when 3 the mea­sure­ment was an­nounced to the sup­port team. Check­ing the raw data, one can in­deed spot a num­ber of un­usu­al prac­tices. For in­stance, some of the guys weren't paus­ing the counter dur­ing the lunch time, as they did be­fore.

  Fig­ure 1 Time spent by the sup­port every week on the tick­ets re­lat­ed to bugs and re­gres­sions in pro­duc­tion.

• We can also track the num­ber of tick­ets opened by the sup­port and as­signed to the de­vel­op­ers, tak­ing only the ones which are closed as solved dur­ing the cur­rent or the next sprint. The fil­ter­ing is im­por­tant here: some tick­ets are not re­al­ly bugs which af­fect cus­tomers di­rect­ly, and they end up be­ing closed months or years lat­er, some­times be­cause no­body cares about them. Such tick­ets would pol­lute the data set, low­er­ing its rep­re­sen­ta­tiv­i­ty.

  The met­ric can­not be gamed eas­i­ly. The in­ter­est­ed par­ty is the de­vel­op­ment team, while the tick­ets are cre­at­ed by the sup­port. De­vel­op­ers don't have lever­age to pre­vent the sup­port from open­ing too many tick­ets. The only thing they can do to es­cape the met­ric is to wait for two sprints and only then fix a giv­en prob­lem. It was un­like­ly that any­one would do such a thing, giv­en that near­ly every one of those tick­ets have a high pri­or­i­ty.

• We are also able to mea­sure the num­ber of hot­fix­es, less is bet­ter. I haven't thought enough of the reper­cus­sions, so I missed the fact that such met­ric would en­cour­age the de­vel­op­ers to merge sev­er­al hot­fix­es into one, in or­der to do less of them. And that's ex­act­ly what they did. Very soon, the met­ric was con­sid­ered harm­ful and aban­doned.

• While the busi­ness didn't know ex­act­ly why cus­tomers were leav­ing, there was al­ready a met­ric mea­sur­ing how many new cus­tomers there were per year, and how much have left. Giv­en the nar­row do­main and the small num­ber of po­ten­tial cus­tomers, it is not sur­pris­ing that the met­ric was mea­sured once per year, and not more fre­quent­ly. A new cus­tomer, or a cus­tomer who left, was a big event for the com­pa­ny.

• An­oth­er met­ric avail­able for us was from the hu­man re­sources. They knew ex­act­ly how many de­vel­op­ers were sick (in­clud­ing burnouts), and for how long. We took care, how­ev­er, to put this val­ue in con­text. Peo­ple of­ten get sick in Oc­to­ber and No­vem­ber; such ex­ter­nal fac­tors should be ac­count­ed, and met­rics should be ad­just­ed ac­cord­ing­ly.

• Fi­nal­ly, a Gantt chart with the dif­fer­ent fea­tures re­leased or be­ing de­vel­oped was cre­at­ed, en­riched with the ex­tra data, show­ing when the com­peti­tor re­leased a sim­i­lar fea­ture. Fig­ure 2 shows a sim­pli­fied ver­sion of such Gantt chart. Green 1 means we're great: we re­leased it first. Red means the com­peti­tor was faster; some­times much faster than us 2. An ad­di­tion­al vi­su­al­iza­tion al­lowed to have a vi­su­al com­par­i­son of the dif­fer­ence in re­lease time for the past year, in or­der to see the trend.

  Fig­ure 2 The Gantt chart, with the em­bed­ded re­lease dates of the fea­ture by the com­peti­tor.

Next, it was time to talk with the de­vel­op­ers. As it was clear that some­thing wrong hap­pens around the num­ber of bugs reach­ing pro­duc­tion, it was still nec­es­sary to find why this would hap­pen.

• Why are there bugs and re­gres­sions? Be­cause we work un­der pres­sure.
• Why do you work un­der pres­sure? Be­cause busi­ness needs fea­tures fast.
• Why do you think the busi­ness needs fea­tures fast? Be­cause there is a com­peti­tor who may re­lease those fea­tures soon­er.
• Why do you men­tion the com­peti­tor in the first place? Be­cause the stake­hold­ers fo­cus on the com­peti­tor when talk­ing about the pri­or­i­ties.
• Why stake­hold­ers do that? Be­cause the com­peti­tor's fea­tures make us lose cus­tomers.

Es­sen­tial­ly, cus­tomers were leav­ing for the com­peti­tor be­cause of the bugs, and the bugs ex­ist­ed in the first place be­cause the cus­tomers were leav­ing for the com­peti­tor. This cir­cu­lar re­la­tion had to be bro­ken at some point.

One im­me­di­ate con­se­quence was that the Gantt chart, es­pe­cial­ly com­par­ing the com­pa­ny with the com­peti­tor, had to be hid­den from the de­vel­op­ers, as it would only make things worse. The only ef­fect of such vi­su­al­iza­tion would be to in­crease the stress, and so in­crease the num­ber of bugs, and slow down de­vel­op­ment—the op­po­site of what we want­ed.

Next, stake­hold­ers were asked to stop talk­ing with the de­vel­op­ers about the com­peti­tor. It has ab­solute­ly no val­ue. I took time to ex­plain to the stake­hold­ers that the fea­tures and the time it takes to de­liv­er those fea­tures is not the only im­por­tant point. For them, it was cru­cial. But com­par­ing their pri­or­i­ties with the con­cerns of the boss showed the dis­crep­an­cy, which forced them to ad­just their pri­or­i­ties ac­cord­ing­ly, putting more fo­cus on sta­bil­i­ty.

Two months lat­er, I came back to this com­pa­ny to see how things are go­ing on. Over the two months, the num­ber of bug tick­ets slight­ly de­creased, but there was noth­ing im­pres­sive. Two sprints per month meant four val­ues, and this wasn't too rep­re­sen­ta­tive. Talk­ing with the team mem­bers, I had an im­pres­sion that noth­ing changed, ac­tu­al­ly. So we made an­oth­er five whys ses­sion. We took a tick­et which was solved the day be­fore. The bug con­sist­ed of a wrong con­ver­sion be­ing used for a giv­en val­ue, which im­pact­ed most of the cus­tomers in a quite crit­i­cal way. The unit tests were un­able to catch the is­sue, be­cause the de­vel­op­er in­tro­duced the very same con­ver­sion er­ror in the tests as well. The prod­uct own­er haven't no­ticed the prob­lem ei­ther, and it was only af­ter a call by an an­gry cus­tomer that the team found that the con­ver­sion is wrong.

• Why did the bug hap­pen in the first place? Be­cause I was ex­haust­ed and haven't no­ticed that I called a wrong method.
• Why no­body else caught up the mis­take? Be­cause there is no pair pro­gram­ming or code re­views.
• Why isn't there pair pro­gram­ming and code re­views? Be­cause we don't have time for that.
• Why don't you have time for that? Be­cause we need to ship quick­ly.
• Why do you need to ship quick­ly? Be­cause there is a com­peti­tor who may re­lease the same fea­ture faster than us.

Here we are, back to the orig­i­nal prob­lem we had two months ear­li­er. The stake­hold­ers were still em­pha­siz­ing the im­por­tance of re­leas­ing ear­ly, de­spite the fact that the boss clear­ly stat­ed what is im­por­tant for the busi­ness. They were still talk­ing about the com­peti­tor to the de­vel­op­ers, in­stead of fo­cus­ing on the num­ber of bugs re­port­ed by the sup­port. This time, in or­der to en­cour­age to change this be­hav­ior, I in­stalled a mon­i­tor in the space where de­vel­op­ers were work­ing, the mon­i­tor show­ing the num­ber of bugs in the cur­rent sprint, a com­par­i­son with the av­er­age for the last ten sprints, and the chart show­ing the evo­lu­tion over a year. This was vi­su­al enough to change the fo­cus.

I also ad­vised the team to start mak­ing code re­views, and cre­at­ed the cor­re­spond­ing met­ric. Af­ter a code re­view, the orig­i­nal au­thor can mark that the re­view­er pro­vid­ed a valu­able feed­back, and can also mark that the re­view­er found a bug. The met­ric would mea­sure the num­ber of those marks per per­son. We also dis­cussed pair pro­gram­ming, but the team was against it, with the ex­cep­tion of the man­ag­er. I ad­vised to for­get about pair pro­gram­ming: man­ag­er's opin­ion is ir­rel­e­vant here. If no­body among the mem­bers want to do it, they won't; if they are forced to do it by the man­age­ment, they'll find a way to make it in­ef­fec­tive.

Re­gard­ing the ex­haus­tion fac­tor, I ad­vised the man­ag­er to stop track­ing how much time peo­ple are pre­sent in the of­fice. I made him agree in front of the team that team mem­bers can leave ear­li­er if they feel tired, and there would be no neg­a­tive con­se­quences. It was es­sen­tial to em­pha­size that they will not be eval­u­at­ed based on the num­ber of hours they would spend in the of­fice. This wasn't an easy task: the cus­tom in French IT in­dus­try is to make the hours spent at work a ma­jor, and some­times the only, mea­sure­ment. I also imag­ined a met­ric which would show whether the ex­haus­tion fac­tor drops over time or not. De­vel­op­ers were asked, once per week, anony­mous­ly, whether they con­sid­er stay­ing in this com­pa­ny for the rest of their life. Few months lat­er, I would be us­ing a very sim­i­lar mea­sure­ment, the Fri­day's mo­ti­va­tion met­ric: “I want so much this week to end!” for an­oth­er team.

Two months lat­er, it's time to check the re­sults. Those re­sults are rather im­pres­sive:

• The de­ploy­ment process is ful­ly au­to­mat­ed, and takes about one minute. Nice.
• Code re­views work great. They helped find­ing two bugs the first month, and four bugs the sec­ond one, that's six bugs which nev­er reached pro­duc­tion.
• The num­ber of bug-re­lat­ed tick­ets cre­at­ed by the sup­port dropped sig­nif­i­cant­ly. I had to spend some time check­ing raw data to find out whether there was gam­i­fi­ca­tion in­volved or not. Seemed that there was no such thing here.
• When I asked the man­ag­er whether there are per­sons who stay less time in the of­fice, he gave me the ex­act an­swer I was hop­ing to hear: “No idea!” This shows that he ac­tu­al­ly haven't tried to check who left the of­fice when, and that's great.
• Much more de­vel­op­ers start­ed con­sid­er­ing that they could stay in this com­pa­ny for the rest of their life. This could be due to the fact that the de­ploy­ment was au­to­mat­ed. Or the fact that there was less stress now. Or the fact that the qual­i­ty of the soft­ware prod­uct in­creased. Or pos­si­bly all those fac­tors com­bined.
• The Gantt chart was nev­er shown since then to the de­vel­op­ers; only the boss could see it. There was noth­ing sig­nif­i­cant there yet, how­ev­er. Nor did the boss have some sig­nif­i­cant data about the new cus­tomers, and the cus­tomers who left.

In or­der to show to the team that they are go­ing in the right di­rec­tion, and en­cour­age them to con­tin­ue, I sug­gest­ed to the man­age­ment to give a fi­nan­cial re­ward to all the mem­bers of the team, and to de­fine a se­ries of oth­er re­wards when the num­ber of bug-re­lat­ed tick­ets would hit spe­cif­ic thresh­olds for the first time for sev­er­al con­sec­u­tive sprints. I also asked to the boss to give his con­grat­u­la­tions to the de­vel­op­er who found the most (that is, three out of six) bugs dur­ing code re­views, as well as to the whole team, who pre­vent­ed six bugs from af­fect­ing the cus­tomers.

While talk­ing with the team, I no­ticed a few con­cerns over the qual­i­ty of the tests. In­deed, there were a bunch of com­plex meth­ods which weren't test­ed well enough. I ad­vised to set up a mon­i­tor show­ing the branch cov­er­age for all meth­ods with com­plex­i­ty high­er than x. That x would be ad­just­ed by the team over time. The orig­i­nal val­ue that I sug­gest­ed, for in­stance, was com­plete­ly wrong, and was ad­just­ed two days lat­er.

Three years lat­er, I con­tact­ed the boss again to see how things are go­ing. The com­pa­ny is do­ing great, un­like the com­peti­tor. There are some im­por­tant new cus­tomers. A num­ber of oth­er cus­tomers left, but not be­cause of the bugs. The boss was un­will­ing to dis­close the real rea­son. A new met­ric ap­peared: the num­ber of de­ploy­ments. There are now about one hun­dred de­ploy­ments per year, with up to four re­leas­es some days. The high fre­quen­cy means that there is no need for hot­fix­es any longer. If there is a bug, it will be solved in the next or­di­nary re­lease. De­vel­op­ers have less health prob­lems, which is ab­solute­ly great. The boss didn't know what's the state of the code cov­er­age, but from the pos­i­tive feed­back he gave about the de­vel­op­ers, I'm sure they got their tests right. The Gantt chart I was so proud of was thrown away: in­stead, time-to-mar­ket (TTM) is now mea­sured. The man­age­ment learned their les­son: the de­vel­op­ment team has no knowl­edge of the ex­is­tence of the TTM met­ric; they don't work un­der pres­sure any longer.

Com­par­ing the sto­ry in this ar­ti­cle with the pre­vi­ous one, this one looks messy. In the Amer­i­can com­pa­ny, Andy have set sim­ple, straight­for­ward met­rics, which were con­sid­ered per­ma­nent. There were no re­vi­sions, no new met­rics over time. On the oth­er hand, the met­rics from the French com­pa­ny were rather com­plex, and changed all the time. Old ones were re­moved. New ones were cre­at­ed. The fol­low­ing il­lus­tra­tion shows the volatil­i­ty of the met­rics. No­tice that none sur­vived un­changed for three years and four months.

Fig­ure 3 The volatil­i­ty of the met­rics over time.

The fact is, the volatil­i­ty is what makes the sys­tem ef­fec­tive. As the sys­tem which is be­ing mea­sured evolves, so should the met­rics. The met­rics in­flu­ence the sys­tem, and the sys­tem in­flu­ences the met­rics. If the met­rics are not ad­just­ed to the evolv­ing sys­tem, there is no cer­tain­ty that those met­rics will have any ben­e­fit, and more of­ten than not, an un­evolv­ing met­ric starts to be ac­tive­ly harm­ful very fast. In the case of Andy, the met­rics were wrong from the be­gin­ning, but this is not his ma­jor mis­take. In the sto­ry above, I did spec­tac­u­lar mis­takes too, by sug­gest­ing to use the met­rics which ap­peared to be harm­ful from the be­gin­ning. How­ev­er, I nev­er wrote those met­rics in stone. We draft­ed them all to­geth­er, and went to check what would hap­pen. As soon as we no­ticed that the met­rics were im­pact­ing the sys­tem neg­a­tive­ly, we ei­ther mod­i­fied them, or got rid of them if they were be­yond sal­va­tion. When it comes to mea­sur­ing the met­rics them­selves, more volatil­i­ty is bet­ter.