Home Home Posts Rants about IT DevOps Stuff I'm working on

Expensive projects are really cool

Arseni Mourzenko
Founder and lead developer, specializing in developer productivity and code quality
100
articles
June 19, 2016

Following my previous article, a colleague told me a similar but much more impressive story which happened to him. Since he doesn't enjoy writing too much, he invited me to tell the story here.

Six years ago, Nicolas was hired as a freelance developer in a French multinational corporation. A year ago, the company started a large internal project which experienced a few problems. Being three months late, the project was considered high priority, and it was decided to increase the number of people working on it, which included hiring a few freelancers such as him. Clearly, Fred Brooks' never wrote anything about that either.

The project's goal was to deal with data provided by the retailers. It consisted of:

  • Communicating with a proprietary French product which collected and organized this data, which appeared to be extremely complicated, since the product was particularly unfriendly—no API, no MQS endpoints, and some sort of proprietary, non-SQL database developed exclusively for this product (and which had a nice characteristic of crashing once per day when used as intended, and much more often when developers were experimenting with it).

  • Doing some statistics stuff with a huge number of businessy-hairy rules which only the actual users could understand, but rarely be able to explain clearly.

  • Presenting the results in a form of interactive charts. The interactions being numerous and complex, the third-party library chosen originally for this task was insufficient, and was originally hacked to change some of its behavior, and then simply replaced by an in-house solution. Nicolas was in charge of one of the parts of this component.

The freelancer found some interest in the project and had an opportunity to talk to a lot of people in this company. One question which bothered him for a long time was how those users were and are doing their jobs, before the project even existed and now, before it is finished. Surprisingly, his boss couldn't answer the question, and neither could the other people working on this project. Everybody claimed that the project is extremely important, but there was no explanation whatsoever about what it will actually change for the end users.

Then, finally, during a coffee break, he met one of those users, Stéphane: it appeared that half of the users worked in the same building, just on a different floor. The person revealed him that the project is indeed promising and would have a huge impact. Right now, those users are limited by the functionality of the product which collects and organizes the data provided by the retailers. There is some statistics stuff in it, and some charting, but both are very limited.

When Nicolas asked whether processing and charting could be done in Excel, Stéphane told that before being hired in this corporation, he was in a company where they indeed used Excel and a bunch of scripts written by an in-house programmer in order to accomplish the same result as they would when the project will be finished.

Interesting... So why wouldn't they use the same approach here? Stéphane suspected that there are technical reasons for that. From there, Nicolas started to explore the first part of the project—the one which communicated with the data collection product. A few discussions with the members of the corresponding team helped making an interesting discovery: while the product made it particularly difficult to interact with, there was a hidden and undocumented feature which allowed the relevant data to be exported as an Excel file. The feature was dismissed by the first team because of the performance penalty: for a sample set of data, it took two seconds to get it from the database, and nearly two minutes to generate the Excel file.

Having generated an Excel file as an example, Nicolas sent one to Stéphane, asking how close the actual format corresponds to what Stéphane used at his previous job. A few minutes later, Stéphane called, asking Nicolas to met him right now. He was very impressed to discover that such spreadsheets exist: while the format was largely different from what he used in his previous job, all the needed data was here, which meant that from such Excel files, the users could do their job right now, without waiting for the project to finish. Not only it made the big project irrelevant for them, but it provided a very flexible way to change the business rules: some could be changed by the users themselves, and others would require just a person knowledgeable in scripting.

Stéphane was so excited that he asked Nicolas to make something which could make it possible for the end users to make Excel exports themselves. Two hours later, Nicolas finished a rogue script which was making possible to those users to specify the needed parameters through a form, and provided them with the resulting Excel file.

Surprisingly, the slowness of the export didn't seem to bother Stéphane: “A different system we use on a daily basis to do a comparable task takes at least an hour; a few minutes to generate an Excel file could easily be qualified as instantly here.”

“However, I wound't recommend talking too much about our stuff around you.”—said Stéphane confidentially—“There is nothing good at telling people that a project which costed hundreds of thousands of dollars to the company is actually useless. There is too much at stake here, and it's too late to stop the project now, the ripple effect would be tremendous and will necessarily backfire at the one who caused it.”

Few days later, Nicolas left the company for personal reasons.

One day, Nicolas received an e-mail from Stéphane, mentioning that someone talked too much and the management discovered the rogue script and the Excel export practice. The script was destroyed, and management explained to the end users that they don't need one: instead, they will be able to use the product which will be released in a matter of weeks.

Nicolas heard that the project was then delayed for six consecutive months, and then closed without being finished. Communication part of the project was making the proprietary database crash more and more frequently; statistics stuff was always wrong; and charting part had too many glitches to be usable. Two weeks before killing the project, the company threw a bunch of money to hire super-expert (i.e. super-expensive) consultants who were expected to fix the mess. They failed (and still got paid). There were a few layoffs, and someone from the top management was fired.

A year later, a second e-mail from Stéphane told that he was leaving the company, and also mentioned that the corporation is starting another large project which will definitively make it possible to do some statistical analysis and draw charts. Stéphane's manager sent an e-mail to the boss of his boss, explaining that it would probably be more cost-effective to use Excel exports, but received no reply. According to rumors, the project was actually finished, although nine months later than expected, but had so many bugs that it was discarded after a few months.