And what if most projects were research projects?
When I started to work on the latest project requiring skills in Linux, Python and Node.js, that is three domains I never used before, I was very clear with myself: the project will take what it will take, maybe two months, maybe six, maybe three years.
While I had the project roadmap and a goal, I had no idea where would it get me precisely. Day after day, the project was moving towards, sometimes backwards as well, and day after day I was facing a risk of being blocked on some issue for weeks or months. And I was, several times, but I couldn't care less. Well, I cared, because it was sometimes exasperating, like my experience with MAAS: after losing three weeks, I was forced to abandon it, since it simply doesn't work. At all.
Without too much Linux knowledge, I obviously found myself in WTF mode more often than on ordinary projects (i.e. the ones where you don't learn anything). Bad things happened, sometimes because of the bugs in Linux applications, often because of my lack of knowledge. Rarely, there were WTF moments like the one with rsyslog:
I was migrating everything to
syslogin order to use a common logging server. It worked well for Apache, since it took me probably less than a quarter of an hour to setup the thing. Quick estimation exercise: if it takes 15 minutes to make Apache usesyslog, how much time would you need to do the same change for Nginx?
In reality, Nginx appeared much more reluctant. I finally discovered that I was using a version which doesn't support logging to
syslogyet, and cannot possibly use a version which does.
It appeared, on the other hand, that
rsyslogis able to monitor files and import data from them. Great. Or maybe not, since hours of attempts led me to a headache, and nothing more.rsyslogwas refusing to log anything from Nginx logs, while accepting to get the data from other files.
I checked everything. Owner/group, permissions,
echoing to the end of the file (since it changes the file date, while Nginx logging doesn't). Finally, I found that changes in every file in the/var/log/nginx/directory are silently ignored byrsyslog, while changes outside the directory (including when Nginx is logging directly to, say,/var/log/nginx-access.log) are taken into account. I don't freaking believe this!
Situations like this make it obvious that it is absolutely impossible to make the project schedule. When a tiny task similar to one which took fifteen minutes takes hours of head-banging, there is no way I can tell that I can deliver the project in four weeks. Or eight months. Or three years.
Many projects are similar, sharing two essential aspects: (1) the roadmap is unclear and (2) shit happens.
The roadmap is unclear
Most projects are inherently unclear. It doesn't matter what the guy who wrote the requirements tells, and it doesn't matter that the customer ensured you that he has a clear idea of the future product. Clear projects require clear methodology that most IT people don't know (the next time your manager will rant about his expertise and the decades he spent in the industry, ask him what is the difference between a functional and non-functional requirement).
It's not that bad. What is bad is that management often handles a project as if the roadmap was clear. A boss may decide that the project will be released with exactly those features in exactly three months, and when, eight months later, the project is still not released, the boss is embarrassed, because he kinda promised something to someone. Or an IT firm may sign a contract telling that the customer will pay $80,000 for a specific project, and when six months later, all of the features requested by the customer have nothing to do with the initial vision, lawyers start to make phone calls to decide who should pay for the mess.
I'm not sure why so many people are disgusted by Agile so much that they are calling themselves Agile while managing the unclear projects like they were perfectly clear.
Shit happens
It doesn't even matter how clear the roadmap is, since the implementation itself is often hard to estimate.
I'm quite skillful in C# and .NET. This also means that my estimates of a one-man project became quite precise over time. Usually, if I tell that it will take me five weeks to develop a web app which has a clear roadmap, it means that in three to five weeks, it will be up and running. But even .NET has its caveats, and I know that whenever the project contains something even weakly related to WCF, I have to be very careful with my estimates, because the project can easily slip by one or two weeks because of WCF configuration nightmare.
I'm not quite skillful in Linux, Python and Node.js, which explains the number of WTF cases I've encountered in the last project.
Not only are there true WTF situations, but also dead ends. I recall working for more than a week setting up a message queue service which would be used, among others, for logging. If only I knew that remote syslog is de facto standard among Linux system administrators! While moving from the message queue service to syslog was a matter of minutes thanks to the decent architecture, the message queue service dead end was still a waste of time in the first place.
The morale here is not that I should have been using what already exists instead of reinventing the wheel. I didn't know how much remote syslog was a standard and how simple and reliable (including, surprisingly, with UDP) it is. Following “don't reinvent the wheel” rule at all costs can lead to a dead end too, by the way: all the time I wasted trying to make MAAS work is a good illustration of that.
Conclusion
When I started to work on the latest project, I was very clear with myself: this is an experimental project, I don't know how much time would it take, and that's clearly not the point. The fact that I had no skills in three main aspects of the project was also an element which encouraged my feeling that this project is exceptional and is made possible exclusively because the person who decides and the person who implements the project are a single person.
But in retrospective, it appears that the project is not that exceptional: most projects I've seen are unclear and made in a context where different events will happen, often making the original estimate completely irrelevant.
The sole difference, after all, is the management adapted to the style of the project. When many managers will expect impossible deadlines from their teams and fire developers in order to show to their bosses that they take seriously enough the problems they created themselves in the first place, I, on the other hand, do what any other manager should be doing: be very clear that there are things which are outside of our control because of the nature of the project, and manage it accordingly.