How to deadlock a project
In 2014, I wrote the article Taking hostages with Waterfall, explaining how Agile moves the power of decision about the scope and the cost of the project from the ones who make the software to the ones who pay for it. For years, I've observed a practice by unscrupulous freelance programmers and software development companies which consisted of giving an unrealistic price/time estimate for a given project in order to get a customer, and then play on the sunk cost fallacy effect to extort money from a customer.
The schema works this way. The customer asks to quote a project. You find that making it would cost from $150,000 to $200,000 in the very best case. You tell the customer that your estimate is $100,000. This way, if there are competitors which are honest, they would give twice the amount, and the customer will choose you, and not them. Since there is no way for the product to be developed for less than $150,000, you spend the $100,000 while being open to any change requests from the customer, and once you reached your original estimate, you ask the customer what to do. You can either give to the customer the source code, given that the product is completely unfinished and cannot work in its current state, or you can continue if the customer agrees to pay more. Since nobody wants to acknowledge that he wasted $100,000 for a useless pieces of code, the customer is likely to tell you to continue. Naturally, if the customer starts blaming you for bad management, you reply that the extra money is explained by the changing requirements. Then, you continue spending money without delivering anything, until the customer refuses to pay. I don't know how things are in the United States, but in France such scam is completely legal, and also very common. My estimate (not backed up by any research) is that 95 to 99% of projects in France happen exactly like this.
The way to escape from this trap is, for a customer, to require the project to be developed in an Agile way. At the end of every sprint, the customer should expect to see a working product, that is, a product which can and possibly is used by the end users. If the freelancer or the IT company is unable to do that, then they shouldn't be paid. It is as simple as that.
But this article is not about customers and freelancers and hostages and sunken cost fallacy. It is about the major mistake I made when developing the blog engine which powers the website you're using right now. In fact, I took a liberty of not using continuous deployment for some time for this project, and created a real management disaster which could lead to a cancellation of the project itself.
I started to work on the blog engine in 2014. Overall, the development was smooth, I was adding features, and I was releasing them as soon as I was confident enough that they work properly. Then, it all changed one day. In 2019, I told myself that the blog engine starts to be quite old. There were a lot of things I disliked about it, and so I needed to perform a lot of changes. Instead of performing small changes, and releasing them step by step, I somehow found myself making large changes one after another, with no releases whatsoever. More changes crept in, and at some point, I was unable to release. The engine I had on my machine was completely different from the engine running on the servers; even database schema was different.
Now, I found myself in a difficult situation. I could throw away all the changes and start over, but sunk cost fallacy effect prevented me from doing it. Or I could continue the changes, hoping one day to be able to migrate the data to the new schema and to finally deploy the new version to production. This is what I was trying for some time, but without seeing the application running out there and being used by real persons, and without even knowing if I'll be able to deploy it one day, the motivation was at its lowest, and I stopped working on the project by the autumn 2019.
Figure 1 Number of commits related to the blog engine.
It is only a few months ago that I forced myself into finishing what I started. This came at a cost. The code base was practically completely rewritten. The structure remains mostly the same, but the individual files have practically nothing to do with the old ones. This is comparable to what would happen if a customer, being kept hostage by a scammer decides to go to another company to rewrite the software. About a month later, the project was in production once again. From now, I can add features or fix issues, and know that the changes can be seen by the end users within minutes.
The fact is, lots and lots of personal projects get abandoned like this. There are mostly two risks which can bring a personal project to a halt. First, the project doesn't seem that interesting, after all, or too complex, or just less fun than a new one. Second, the project deadlock. For a project that just starts, the first risk is more important. I have hundreds of projects I've started, and then never came back to them. It is not a bad thing in itself—developers need to be able to play with some ideas, without having to commit to anything. Naturally, some developers tend to jump from project to project, never finishing anything; this is indeed problematic. As the project becomes more mature, the importance of the second risk grows. Mature projects are mature because of one thing: they are useful. This means that there are chances for a project to be maintained, even if maintaining it is not particularly fun. Deadlocks, however, can happen independently of the usefulness of the project, and they can kill even a project which is very valuable.
The solution is, you guessed, continuous deployment. From the very early stage of a project, you ensure it can be used to accomplish a specific need, and keeping an eye on the time-to-market (TTM) metric (that is, the time it takes for a feature to be developed and released) could help.
The thing is, this solution works well in a corporate environment. The guy who pays the bills is usually not the guy who codes, and that's great. This is great specifically because nobody cares if you want to release or not: the stakeholder requires the features to be released fast, and if you don't do it, you're fired. That's it. For personal projects, this doesn't work this way. One wants to be disciplined, and one wants to tell his friends that he's releasing often, but the fact is that it can also be extremely tempting to try to push a heavy change or an interesting feature, without bothering about the deployment and feature toggles and data migration. And when you're focusing on the exact problem, you can miss the fact that your TTM goes from an hour to several days, then weeks, then months.
Honestly, I don't know what would be a real solution for personal projects. Maybe better metrics could help. Maybe something needs to automatically throw away the changes if a given TTM is reached. Do you have a solution?