Home Home Posts Rants about IT DevOps Stuff I'm working on

Ratings in a CV

Arseni Mourzenko
Founder and lead developer, specializing in developer productivity and code quality
November 14, 2017

Heard today a discussion of two colleagues about a possible improvement of a CV by adding for every technology a rating, showing the mastery level of the person in this particular technology.

I'm sorry about those colleagues. They have a deep misunderstanding of the hiring process and of the value and the form of data. This idea is totally flawed; it's broken beyond repair.

Before explaining why is it wrong to rate one's own mastery of a technology (or to ask a person to rate himself on a CV), let me highlight one of the issues I highlight repeatedly over the past five years: the overemphasis on technologies. It's very simple to explain. There is no such a thing as a Java software engineer, or a Ruby software engineer. There are Visual Basic programmers and there are PHP coders. If you are looking for a code monkey whose job is to produce lines of code, good for you; keep focusing on technologies when reading a CV. On the other hand, if you are trying to hire a person whose job would be to produce value through information technology, individual technologies shouldn't care that much. Great software engineers have a great learning capacity; a professional software engineer who never wrote a line of code in Python could become quite fluent in this language in a matter of weeks. However, a mediocre Python coder would require years of constant effort to achieve the level of a professional software engineer.

This means that the list of technologies shouldn't the main part of the CV. It is still necessary to put this list on a CV, but the emphasis should rather be on the things that actually matter. This makes the discussion about the improvement of that list not that important. But since we're here, let's discuss.

Personally, I'm strongly convinced that I have zero out of five in Java, four out of five in C#, three out of five in Python. What does it mean? Seriously, what do you know about me that you didn't knew before? How could you use this new data if you were about to decide whether you want to hire me or not?

Imagine you are looking for people to work on one of the three projects. One is written in Java, another one in C#, and another one in Python. What would you suggest me if you were hiring me, based on the information above? C# project? Failed. What you didn't know is that I have a heavy preference for Python, and I want to learn Java more.

Different scenario. You're looking for someone to work on a C# project. You have my CV, but also a CV of another person. His profile seems very close to mine, but he rated himself five out of five for C#. Who would be your primary choice based exclusively on the CV? And here, you failed again, and you made the whole system fail. By preferring the one who rated himself higher, you don't take in account the Dunning-Kruger effect. It happened that in the past, I knew C# damn well, much more than any of my colleagues. Since I haven't worked too much with Microsoft's technologies since then, my level decreased, while the loss is recoverable. Moreover, if I put five-star rating to Jon Skeet, how could I possibly put myself more than three stars? But if I put three stars, people from human resources would filter my CV, so I should artificially increase the number of stars, just to avoid being filtered. That other guy did the same thing; however, he didn't use Jon Skeet factor, so he went from four to five stars, even if objectively, his level is lower then mine. See how miserable it gets?

The terrible thing about those ratings is that they present subjective data as if it was objective. By doing that, they mislead the reader.

Staring system is all about numbers. 4.7/5 is slightly better than 4.6/5, which is much better than 2.1/5. This works pretty well as soon as statistics and big data sets are involved. On Netflix, if a movie is rated 4.7/5 by twenty thousand users, it should be pretty popular. If it's rated 2.1/5 by twenty thousand users, something is not right with this movie. This doesn't mean that I will enjoy the first movie and will hate the second one—this is plain misinterpretation of statistics—but it has an actual, objective meaning in terms of popularity, made by averaging individual ratings of a large number of users.

When the data set decreases, the objectivity decreases as well. Imagine the same Netflix ratings based not on twenty thousand users, but only three persons. The relevance of such rating, and essentially its value, would be close to zero.

Back to CV, the ratings we get here are based on the judgement of a single person. This essentially means that statistically speaking, the objectivity is zero, even if one excludes the artificial change of ratings based on personal considerations of the person, such as the artificial increase discussed several paragraphs above. However, presented in a form of ratings, the information makes look itself as something as objective as Netflix movie ratings.

Providing misleading information to attract the attention on less important parts of the CV? I don't see the point, unless it's done to distract from the skills that really matter—one of those skills being the ability to present information correctly.

If self-evaluation there is, it should be in a form which shows subjectivity. “Poor,” “Average,” “Good,” “Great” four-strata system is much better in this way. It shows what it shows: that the person estimates his skill level in PHP as “Average” and his level in JavaScript as “Great.” There is no attempt to pass subjective information as objective here. In this sense, “expert” term should be avoided, because of Malcolm Gladwell's 10,000 hour rule. Here once again, the term has an objective connotation, with a precise metric associated with it.

Let's pretend nobody will be fooled by the form of the data, and everybody understands that those ratings are only indicative of how skillful the person thinks to be in a language. Let's pretend that there were no artificial adjustments either. Now, wouldn't it be really valuable to know what the person thinks of his own skills? OK, let's see. My C# skills as seen by myself:

“For a few years, I was much skillful in C# than all my colleagues. One element which makes me think that is the fact that I ensured numerous training sessions in practically every project, teaching both the basics of the language and the framework, including functional programming aspects, as well as more advanced stuff, such as formal proof through Code contracts. Another element is that I always found myself either directly working or actively helping others on the most challenging parts of the project, where technical expertise in multiple aspects such as IL optimizations was required. In the last few years, I abandoned C# for my in-house projects, and used it professionally only on simple, non-challenging projects, which means that I forgot a bunch of things I knew before, and I'm completely unaware of all the new things which appeared in C# and .NET Framework in those past years.”

This is as short as it gets, and it's a perfectly fine for an interview. I have no idea how this amount of information could be transformed in a number from zero to five, while still remaining any value.

A funny exercise is to generalize the rating system to include first the non-languages, and then non-technologies as well. Non-languages are all the technologies which are not the programming languages, such as LDAP or AWS. Could you rate yourself on microservices? On DevOps? On scalability? As for the non-technologies, they include all the skills required from a developer, but which are not related to a specific technology. For instance, could you give yourself a rating at communication with pairs? Or the understanding of the business domain? Or the ability to use lateral thinking to come with out of the box solutions to a problem?

Back to languages, the funny thing is that it's not that difficult to provide objective, meaningful information. A chart with the number of changed LOC in a given language over a period of ten years, compared to similar charts for other languages, tells so much more than some stupid rating one puts on a CV, based on personal expectation of some outcome which may not even be true. We live at an age of data; do you a service, and start exploiting this data accordingly instead of providing wrong information through wrong methods.