Most performance questions are wrong

Arseni Mourzenko
Founder and lead developer
March 15, 2015
Tags: performance 13 stack-exchange 4 profiling 4

I don't re­mem­ber any ques­tion post­ed on Soft­wa­reEngi­neer­ing.SE re­lat­ed to per­for­mance which would be a good, on-top­ic ques­tion.

I re­mem­ber ques­tions which were com­plete­ly wrong or sim­ply unan­swer­able.

It seems like per­for­mance-re­lat­ed dis­cus­sions are pure­ly spec­u­la­tive. They could be very con­crete, but it re­quires great ef­forts and a lot of ex­per­i­men­ta­tion.

Ques­tions asked too soon

In gen­er­al, peo­ple start ask­ing ques­tions too soon. The worst case is when a per­son starts both­er­ing be­fore even hit­ting the per­for­mance prob­lem. This usu­al­ly takes two forms: how to make a giv­en (un­pro­filed) code faster and which ap­proach to chose.

Both sit­u­a­tions lead to a para­dox­i­cal sit­u­a­tion where we be­lieve that mak­ing a choice right now is easy then post­pon­ing it and deal­ing with the con­se­quences lat­er, but by deal­ing with it right now, we just shift our fo­cus from im­por­tant things to stuff we can't pre­dict.

Or can we? This leads to a prob­lem of op­ti­miza­tion tech­niques knowl­edge, which is a dif­fer­ent kind of ques­tions on Pro­gram­mers.SE.

Op­ti­miza­tion tech­niques knowl­edge

Imag­ine cre­at­ing a C# ap­pli­ca­tion which deals with pic­tures do­ing pix­el-based stuff such as a diff be­tween two im­ages for mo­tion de­tec­tion. Also imag­ine that there are no li­braries which do the thing, so we need to do it from scratch.

  1. One way is to ac­cess pix­els us­ing .NET Frame­work's GetPixel() method. Sim­ple and straight­for­ward, this ap­proach will quick­ly re­sult in ter­ri­ble per­for­mance.

  2. An­oth­er way is to use unsafe and deal with point­ers. Very scary, but still easy and much faster.

  3. Fi­nal­ly, the most bru­tal way is to del­e­gate the job to a GPU, ob­tain­ing ex­cel­lent per­for­mance but at a cost of much greater ef­fort from an or­di­nary C# pro­gram­mer.

Choos­ing the last op­tion looks like pre­ma­ture op­ti­miza­tion: why us­ing an over­ly com­pli­cat­ed so­lu­tion when the im­age ma­nip­u­la­tion may not even be the bot­tle­neck?

On the oth­er hand, the first two al­ter­na­tives are in­ter­est­ing. It doesn't take much ef­fort im­ple­ment­ing the sec­ond one from the be­gin­ning, and it is like­ly that the first one will in­deed be­come the bot­tle­neck. Is there pre­ma­ture op­ti­miza­tion?

I'm hes­i­tant. If I work with a C# pro­gram­mer who wants to write such ap­pli­ca­tion, I would ad­vise him to use point­ers from the be­gin­ning. The real per­for­mance ben­e­fit is much more im­por­tant com­pared to the slight­ly more com­pli­cat­ed code, even in a con­text where the bot­tle­neck might be some­where else.

But this is not that im­por­tant. Imag­ine a pro­gram­mer who doesn't know that GetPixel() is in­her­ent­ly slow. So what? He tries the first so­lu­tion, hits a per­for­mance is­sue, pro­files his ap­pli­ca­tion and im­me­di­ate­ly sees that the prob­lem is GetPixel(). The next half an hour, he spends re­plac­ing the cur­rent code by the one which uses point­ers, the ben­e­fit be­ing that he learnt some­thing in­ter­est­ing to­day (and learn­ing through prac­tic­ing is much more in­ter­est­ing than learn­ing from some­one). The waste of 30 min­utes is not the most ter­ri­ble thing which can hap­pen to a pro­ject.

On Pro­gram­mers.SE, ques­tions of this type are prob­lem­at­ic for two rea­sons:

Ex­plor­ing op­ti­miza­tion tech­niques

The last cat­e­go­ry of per­for­mance ques­tions con­sists of ask­ing not which thing is faster or how to make it faster, but why is one thing faster than an­oth­er one.

Those dis­cus­sions are usu­al­ly miss­ing con­text or don't go deep in the im­ple­men­ta­tion de­tails. For ex­am­ple, a ques­tion may ask about an op­ti­miza­tion tech­nique in Java, while for­get­ting to spec­i­fy the ver­sion of Java com­pil­er, as well as the op­tions. Dif­fer­ent ver­sions may have dif­fer­ent op­ti­miza­tions, and dif­fer­ent op­tions may af­fect per­for­mance as well.

A rel­e­vant ques­tion will list all the specifics (lan­guage ver­sion, com­pil­er ver­sion, com­pil­er op­tions, plat­form, ar­chi­tec­ture and how bench­marks were done) and a rel­e­vant an­swer will ex­plain, based on the source code of the com­pil­er or its of­fi­cial doc­u­men­ta­tion or some re­li­able sources, why is some piece of code faster than an­oth­er one. This is not even close to what hap­pens on Soft­wa­reEngi­neer­ing.SE.