Home Home Posts Rants about IT DevOps Stuff I'm working on

Quantifying time and cost: why do we always get it wrong?

Arseni Mourzenko
Founder and lead developer, specializing in developer productivity and code quality
August 29, 2012
I have an article. I want to publish it as a web page and let the users comment it.
Does this article need to be editable?
No, I have no intent to modify the article from now.
Tell me more about the comments. How are they? Could they be edited or removed?
Those are ordinary comments, with just a name of the author and the body. Once a person sends a comment, this person cannot modify nor remove it any longer. All the comments are displayed as a long list, from the oldest to the most recent one.
What about the visual aspects?
You do whatever you want. I trust your graphical skills. Keep it easy, I don’t want you to spend too much time on it.
Do you have quote?
Let’s say $150.
WHAT?! $150 for this? It’s just a simple page with some comments. There is really nothing complicated.
OK, maybe $100.
I have quotes for $30 from PHP developers.
Well, I can do it for $75.

Discussions like this happen all the time when you work in IT. You are given a summary which barely describe the project, you have no time thinking about it, and then you receive a strong pressure from your boss because some PHP code monkey from a doubtful country is ready to do low quality work for a low amount of money.

How do those three aspects increase the risk of wrongly quantifying both time and cost? How to avoid them? I’ll try to ask all those questions and illustrate all this by making live a web app.

Part 1: embracing the requirements

There is a rule which works well for freelancers: when creating an application, forget about doing things in order, and present as soon as possible something to show, in order to convince your customer that you’re working hard on the project. Telling that you spent the last two months doing the architecture is interpreted by most people as: “I spent two months procrastinating and instead of apologizing for my laziness, I’ll rather tell you lots of technical words you don’t understand.” Instead, a glance on something excitingly visual and graphical convinces your customer ways more, even if you’ve spent a few hours doing the visual.

From this point of view, we, people working in IT, are not very different from our customers. When presented with a short description of a project, we are thinking about the cool features and how nice it would be to code them, and are completely omitting everything outside the description.

We rarely anticipate every smallest part of a project. In most cases, we don’t anticipate the most parts. Some requirements are so obvious that we can’t omit them, but we still frequently forget about every other requirement which is not directly visible.

Illustration: how simple it could be?

As an illustration, let’s imagine the easiest web application we can. It’s just a web page with some predefined text which never changes; people can come and add comments which are then displayed to everyone from the oldest to the most recent one. They can’t register. They can’t remove or edit their comments. Comments are just plain text with a name of the author as a separate field.

Heck, I can do it in less than half an hour! It’s so easy!

What we forgot

Actually, the project is not easy at all. It feels easy, because we omitted to ask ourselves the most important questions:

  • What will happen if a bot starts to post Viagra ads twice per second?

  • What if a user leaves one of the fields empty?

  • What if a user leaves an inflammatory comment? Would the only way be to remove it by manually querying the database? If the user continuously insults all other users, wouldn’t it be possible to block this user by IP?

  • If a user submits a form with a comment being 500 000 characters long, how it would be displayed? What if the user’s name is 500 000 characters long?

  • Are Chinese characters submitted and displayed correctly? What about backslash? What about reserved HTML characters?

  • Is the page compatible with Opera? IE7? Safari? How is it displayed on a mobile device?

  • If the same page is requested 50 times per second, will it cause a DOS?

  • Isn’t there a way to cache the comments (and even the whole page) instead of querying the database at every request?

  • If the database is down, what would happen?

  • In case of an error, will this error, would it be a business layer exception or an SQL error, be displayed directly to the user, instead of showing a nice error page apologizing that something got wrong?

  • Are exceptions logged?

  • How the comments in the database are backed up? What happens if there is a disaster which affects the server and causes a loss of server’s data?

  • Is deployment process automated?

  • If there are four thousand comments, how long would it take to display a page? Isn’t it a good idea to implement pagination, or, better, populate older comments only on demand, having something similar to Google Images?

Who should care?

You may wonder why you have to care. You were given the basic description of the project. The customer never asked you to log exceptions, make a disaster recovery plan or handle the case where one of the fields of the form is empty. In fact, it’s not to the customer to think about it. He doesn’t have to know about disaster recovery plans, or about audit, or why PBKDF2 must be used instead of MD5 to store passwords. He solely expresses his business needs, because his area of expertise is his business, not the way of storing passwords or displaying large amount of comments on a page.

If you simply limit yourself at translating business needs into a more technical language, you are omitting the most important part of the project when quantifying time and cost. You then agree with your customer for $75, while the real cost is at least $800, i.e. ten times more.

There are requirements which are mostly explicit. If the stakeholder tells that an app must let the user to remove the last file the user created through this app, it is clear that one of the requirements is to do exactly that. It is less obvious that you should also have a requirement explaining how the app should behave if the user haven’t created any document yet, or if the document is in use, or if it was already removed, or is read only, or if it is on a flash drive which was unplugged meanwhile. Would a button be available for the removal? Is there a keyboard shortcut for it? Is there a warning that the document will be lost forever, or is there instead an undo feature?

Before giving a quote, you should walk through all those requirements, not just the obvious ones. When you don’t know the answer (for example in a previous illustration, do you need a warning or an undo), you should ask the stakeholder first.

Part 2: taking time to think

In the previous part, I’ve shown how easy it is to be fooled by sparse business requirements and to quantify time and cost more than ten times lower than the actual one.

By more time, I mean, much more. When it comes to modifying a road for example, calls for bids made by the State contain hundreds of pages to read, describing precisely the context, what to do, what to wear when doing it, etc. A considerable amount of time is needed to write those documents, and some amount of time is required in order to read and understand them.

You certainly don’t need to write hundreds of pages when you’re requested to add the most basic commenting feature to a static web page. Still, some preliminary thinking is required in order to avoid the surprises later during the implementation.

More importantly, you need time to think. Some requirements are hard to spot, and are discovered only after some elaboration. More you understand the business problem, more requirements you can think about, and more requirements you think about, more others you can discover.

Elaborate at least some documentation

Documentation is crucial even at the early stages. You can hold a few business requirements in your head, but when you start thinking about the details, it is impossible to remember every aspect of the project.

It would be too cost prohibitive to create WBS, Gantt charts, a full list of functional and non-functional requirements and similar documents for every submitted project. It is simply impossible given that:

  • Your boss (or your customer) is unwilling to wait for your answer too long,

  • Not every idea is brilliant, and you may receive lots of ideas for projects which will never be ever at planning phase,

  • Initiation phase is done in collaboration with the stakeholders, and often require the involvement of many people with diverse areas of expertise,

  • All this time will be lost (and probably unpaid) if the project is halted at this early stage.

Still, some lightweight documentation may be only beneficial. I distinguish three types of information which may be gathered at this level:

  • Business requirements. This includes everything the stakeholder can communicate you about the project itself from the business point of view.

  • Technical data. This is more about technical aspects which doesn’t concern the stakeholders, but are critical to quantify time and cost. For example, if the project contains some dark parts and requires a few prototypes, this should be described in this part of the documentation.

  • Information for stakeholders. This has a very specific purpose of training the stakeholders or explaining something about the project or the overall process. For example, if the stakeholder is imagining that something is possible, while given the specific constraints, it’s not, this would be in this category.

Note that if done correctly, this documentation may be reused later. As a freelancer, I often have to work on projects where the only documentation is the project summary of several paragraphs; not pages—paragraphs. Those projects are often of the same type: they waste peoples’ time and money, and they end up in a terrible product nobody can or would like to use.

Taking a few days actually thinking about the project and writing the thoughts down in an organized way would probably help a lot in those cases; for some of those projects, thinking about them for a few hours would be enough to understand how useless they are.

Ask questions

There are chances that the customer doesn’t have a clear understanding of the business requirements themselves. If it is the case, you should help them deepen their project by asking questions and providing basic technical assistance. For example, if you take the project from the part 1, a possible dialog would be:

You told that all the comments are displayed on the same page. Am I right?
What happens if there are many comments? I mean a huge quantity, like several thousands.
You still display them on the same page.
Well, you see, it may cause some performance issues. Moreover, the page would be too big in size, so downloading all those comments at once would be painful on a device with a slow internet access.
What do you suggest?
We may want to implement pagination. This is the basic form, and is simple to do. Users will be able to change pages to view older comments. Tell me, how users will use those comments? Would they be searching them randomly, i.e. be interested in being able to access for example page 9 out of 14, or they would rather read the most recent ones, and then go to the older ones if need?
I think they will actually read the newest ones only, and maybe then go and read the previous comments.
OK. In this case, it would be a better alternative to avoid pagination, and to let the user to display a certain amount of older comments at a time.

Before asking the question, it was impossible to know if pagination is a good idea. It was so unclear that the only requirement issued from the original description of the project was to display the comments from the oldest to the newest one. After this dialog, it appears that the comments must be displayed in the exact opposite order, that not all of them should be displayed, and that pagination is not an option.

Encourage prototyping

Basic documentation and a plethora of questions still isn’t enough to be able to quantify precisely time and cost of a project. It would definitely help in the basic example I gave in part 1, but would not work for medium or large scale projects.

Agile methods help a lot in this case. If you can’t possibly know how much will it cost to finish the whole project, you can still give a precise quote for a small part of it. Creating a few prototypes is a good idea too, since not only you will be able to quote more precisely a small prototype than the whole app, but also prototypes are themselves done to be able to know more precisely the overall cost of a project by eliminating the unknown from the most obscure parts.

From the point of view of cost and time evaluation, prototypes and agile methods are very different things. As previously said, prototypes tend to bring to light the parts which are unclear. Prototyping helps rapidly identifying the problem and the solution (or the absence of it), without wasting too much time writing clean code, designing clean architecture, etc. Some tasks just cannot be evaluated, or, in other words, in order to know how difficult they are, you need to do them in the first place. Prototypes are used precisely in those cases.

Agile methods, on the other hand, are simply separating a large project into smaller chunks to be able to do each chunk separately, receive feedback from the stakeholders, and continue with the remaining chunks. This makes the overall process very fluid and progressive, and allows the stakeholders not only to see the current state of the project and its cost, but also to stop the project if the cost becomes too high, but still have something functional.

Admit that you don’t know everything

If you’re a developer, you might be able to evaluate the cost of the development itself, but that’s all. It’s up to the DBA to determine the cost of designing the database, or to the interactivity designer to determine how much would it cost to work on interactivity design for a specific application.

Moreover, you as a developer rely on other people. If the DBA creates a complex design for a database, it may affect your code, requiring more work to do, which translates into more time and higher cost. If the graphic artist creates fancy graphics for a website, it would require more complicated HTML and CSS, so higher cost.

Part 3: avoiding being influenced by stakeholders

Misunderstanding as the root cause

Your boss is interested in low costs. While he understands that he gets what he pays for, he finds the business requirements pretty simple and will not understand why the corresponding price is so abusively high.

This means that from the beginning, there is a sort of misunderstanding.

Recently, I had to work on a project of a web app. The customer is a programmer who had an idea and tried to make it concrete through code, with intent to create a commercial-grade app used all around the world. The most surprising part is that this programmer doesn’t seem to understand how much effort should be involved in a commercial-grade app targeting international market, and all the efforts of the project are concentrated on programming itself. Graphics are horrible. UX is nonexistent. The interface is translated in English by a French person who doesn’t know English very well. The database schema makes me cry (a phone number is saved in an int field, while a digit between one and six is stored as nchar(10); there are no constraints; every aspect of the schema is a candidate for The Daily WTF). Nobody ever profiled any part of the app, and given that any page is making at least twenty queries generated by Entity Framework (you know, the ones which don’t fit in a screen and are totally unreadable), SQL profiling is out of question. There is no cache at all. Passwords are stored in plain text. The logging is nonexistent. There is no documentation. The only comments I’ve seen are either unnecessary or wrong. When I asked how this person rates the code base on a scale of 1 to 10, the person answered “8”. Actually, it would be 2, not 1, because there are some patterns used in the code.

In short, the actual work represents at best 1% of the work required to do something correct. Sadly, this is the case of most IT projects.

Being a not very experienced programmer, this person has a deep misunderstanding of every other area involved in a software product. For example, he probably don’t know what interaction design is about.

A business person has a similar vision of the development part of the software product, but, see, more limited. Development is so unclear for most people that they would imagine that it’s something complex, but they never imagine how actually complex it is in reality.

When it comes to imagining the product they need, they imagine only the direct cost of implementing the business requirement. It’s actually easier to illustrate this by an example of a change of a product.

Let’s say you’ve developed a software product which deals with some data stored in a certain way. Later, a stakeholder asks you to make a tiny little change in this data; in reality, it requires you to change all the design, all the way you store your data, a large part of the database schema, and do some minor UI changes. You tell that the actual cost of such a change is $5 000, while the stakeholder expected at most $40.

The same applies to new products. Since the stakeholder cannot possibly foresee most of the functional and non-functional requirements, he would be surprised by the abusively high cost you give to him, and try to convince you that the cost should be lower.

Incompetence as the cause of misunderstanding

I enjoy downloading and reading the documents in calls for bids published by the State administration. What I enjoy the most is the fact that for the tiniest project, there are hundreds of pages describing until the tiniest detail what is the work to do, and all those pages are written by very experienced people who know extremely well what they are writing about. They spent years in college learning how to do this stuff, and they have a huge professional experience.

I try to imagine something similar to what happens in IT world:

State representative
I have a bridge over a river and it’s too narrow. I want it larger. Could you do it the next week?
Wait, I need some more details…
State representative
Well, it’s a bridge with two lanes, and I need four instead. How much would it cost?
I really don’t know…
State representative
Come on! It’s a simple bridge! There is nothing difficult. The bridge is already built; you have just to enlarge it. Do you think $50 000 is enough?
You should maybe provide a detailed description…
State representative
I already told you precisely what to do.
In all cases, in order to make a bridge larger, it would require a lot of work, so $50 000 is hardly enough even for a small one.
State representative
Seriously?! How much would it cost to add two more lanes?
Hm, well, I don’t know… What is the bridge you’re talking about?
State representative
The one which connects La Rochelle to Île-de-Ré¹, it’s just 2.9 km long. Nothing complicated. I’m sure you can do it the next week.

The photo of  the 2.9 km long bridge to Île-de-Ré

Handling the misunderstanding

Explaining why your quote is ten times higher to the expectations of your boss is hard. Showing to your boss some of the requirements he didn’t think about and telling that there are more may help, but is not very effective in practice, and requires drafting the requirements, which takes too much time.

Taking an example in a domain other than IT, like the illustration above, may have an impact in some cases, but is completely ineffective in others. Moreover, business people are rarely inclined to read discussion-oriented stuff. They simply don’t care.

The way I’ve found which seems to be more effective is to have a review process where you ask all questions which come into your mind. They don’t have to be answerable by the stakeholder; they don’t even have to be answerable at all.

In Florence² project, the project starts by the customer submitting any document she has that is related to the project. Then, before starting to draft the backlog and before even accepting or rejecting the project, any person of the company, interested in this project, can ask questions. The intent is to obtain enough information from the customer, but this part can be enlarged to unanswerable or technical questions which have the side purpose to show to the customer the real work to do.

1 The original cost of the Île-de-Ré bridge is 58.7 millions of Euros. It is the second longest bridge in France.

2 Florence is the customers’ management software for freelancers and software development companies currently in development by Pelican Design & Development.