Interviewing ChatGPT for a job
Recently, I was reviewing the interview questions for a few companies. Many of them were unfortunately only about algorithms—sadly indeed, since those questions were for the jobs that have nothing to do with algorithms, and if the interview process is evaluating the skills that are not needed for the job, and not evaluating the skills that are actually needed, well, bad things will happen.
Reviewing those questions, it made me wondering how ChatGPT would solve them: my theory was that it would score pretty high. A little experiment confirmed that. Then I started to think about the questions I ask to the candidates during the interviews, and what if I asked those questions to the artificial intelligence. ChatGPT surprised me in that it was giving very complete responses, outperforming me (and the candidates I had in the past) more than once. Sometimes, however, it wasn’t “thinking” largely enough. Here’s one of the questions I asked:
Imagine a banking application that stores issuers—the identifiers of companies. Each issuer has six symbols, and each symbol is either a capital letter, or a digit (so there are thirty-six possible values for each of the six symbols). You can store it in a form of a
char(6)in a relational database, but your colleague tells that this wouldn’t be optimal in terms of space usage, and suggests an optimization. After all, every combination—and there are 2,176,782,336 possible values—can be stored in 32 bits, whereas
char(6)is 48 bits. What would you reply to your colleague?
I asked this question to a human candidate once. The response was missing the point. The candidate started to search for a way to do such storage optimization, and then lost the track of his own thoughts—I would be lost too if I were him. What he missed is that I was evaluating his pragmatism. The candidate could have answered that this is a sort of premature optimization, and, moreover, it doesn’t look like there would be any noticeable impact on the database here. Making any complicated changes in the format of the storage would only harm the project, as it would make the code more complex, with no clear benefit. Finally, the candidate could be mentioning the fact that databases usually provide transparent compression that can be enabled for a given column.
ChatGPT answered this:
Your colleague’s suggestion to use a more compact representation for the issuers in the banking application is valid in terms of space optimization. Storing the issuers using 32 bits instead of 48 bits can save storage space, especially when dealing with a large number of records. However, there are other factors to consider before deciding on the most appropriate approach. Here are some points to address in your reply:
- Readability and Maintainability: Storing the issuer as a
char(6)provides human-readable and self-explanatory information about the issuer. Developers and database administrators can easily understand and work with the data, as the characters represent letters and digits. Using a compact binary representation may sacrifice readability and make it more challenging to interpret the data at a glance.
- Indexing and Query Performance: If the issuers are frequently used in queries, having a compact binary representation could impact indexing and query performance. With a
char(6), the database can utilize indexing structures efficiently, making it easier to search, sort, and filter by issuer values. Binary representations would require additional conversions or bitwise operations to perform such operations, potentially affecting performance.
- Application Logic and Integration: Consider the impact on the application logic and integration points. If other systems or components interact with the banking application, they may expect and depend on the
char(6)representation. Changing the representation to a compact binary form would require modifications in various parts of the application and any integrations, potentially introducing complexity and compatibility issues.
- Scalability and Future Requirements: Assess the scalability and potential future requirements of the banking application. While the space optimization gained from using a compact binary representation might be beneficial in the short term, it’s essential to consider future needs. If the application is expected to evolve, accommodate additional features, or expand the issuer data with additional attributes, the compact representation might become less advantageous over time.
- Storage Costs: Evaluate the overall impact of storage costs in the context of the banking application. While the difference in space usage between
char(6)and a compact binary representation might seem significant on an individual record level, the actual impact on storage costs depends on the scale of the application, the number of records, and the storage infrastructure in place. In some cases, the potential savings in storage costs may not justify the trade-offs mentioned above.
In conclusion, while the idea of using a compact binary representation for storing issuers in the banking application is intriguing in terms of space optimization, it’s important to consider factors such as readability, maintainability, query performance, application logic, scalability, and storage costs. It would be worthwhile to have a discussion with your colleague, taking these aspects into account, and make a decision that aligns with the specific needs and priorities of the banking application.
While pedantically verbose, the answer haven’t mentioned transparent compression, but remained focused on the narrow context of different representations of the same data. This sort of “tunnel vision” became much more apparent with another question:
Imagine your client wants to launch a new project, and wants to know if it would be better to use PostgreSQL or MongoDB. What should be taken in consideration to make the decision?
This question looks simple, but it’s actually tricky. There are essentially three possible answers, creating three groups of candidates.
The first group would start explaining that MongoDB is better. Most candidates I interview, especially for programmer jobs, are in this category. They have a series of idées reçues about the databases, one of them being that NoSQL is inherently better than “old legacy stuff,” and that it is inherently scalable, by opposition to Oracle or to Microsoft SQL Server or to virtually any database that has SQL in its name, that, according to them, cannot scale.
The second group has a good grasp of relational databases and other types of databases as well. They can provide a pretty objective comparison of pros and cons, and they put the emphasis—when answering this particular question—on the data structure that is being used by the application, i.e. can it be represented as a series of tables, or as a series of documents. ChatGPT is in this category. It talked about data model, scalability, performance, schema flexibility, ecosystem and tooling (I’m impressed!), ACID, deployment and operations, community and support, and finally cost considerations. With all this information, ChatGPT outperforms any candidate I had until now. But there is actually a third group...
The third group doesn’t think about the problem “the right way.” In fact, people from this category think differently, and especially they think outside the scope. While they can do an actual comparison, like the people from the second category, they can also respond with a question. Such as, why do we need to choose a database in the first place. And that’s the point: maybe we do, or maybe we don’t. Maybe the application is structured in a way that it is not fitted for PostgreSQL, but also not fitted for MongoDB. Maybe it should use both—part of data is ideally structured as a series of tables, and another part—as a series of documents. Or maybe none of the two options fit. I can think of at least two examples where this is the case: either the project doesn’t deserve a database at all (and I did have projects that were storing data in plain files, and it worked exceptionally well), or it needs something more specific, such as InfluxDB.
What’s so valuable with the persons from the third category is that they bring an insight that could completely change the picture, and save a lot of money. Imagine the case where, indeed, InfluxDB is the answer. But it wasn’t considered. Why was it missed? Easy. The project manager had a talk with an ops guy, who suggested PostgreSQL and MongoDB, as this is what ops were familiar with. The guy wasn’t even considering other options, because this is outside his area of expertise. The manager wasn’t considering the other options neither—that makes perfect sense. He went to his team with a preconceived idea that there are only two options, and therefore asked a straightforward question: which one, of the two, should we take. The team picked the least bad option, and proceeded with the development, spending extra time because the choice they made was not optimal. Once the application was released in production, this choice caused it to perform poorly in terms of performance, and waste extra disk space, causing additional expense to the company. So much waste, just because right questions were not asked early enough!
As I said, ChatGPT outperformed all the candidates in terms of the ability to objectively compare PostgreSQL to MongoDB. It surely have read lots of papers on the subject, lots of articles, lots of online discussions, and it was able to process all this information, and output it in an impressive form of a relatively short answer that gets to the point. That’s great. But we, humans, still have an edge over ChatGPT. Unlike artificial intelligence, we could think outside the box, and, given a question, say to ourselves: “wait, that’s not the right question—I smell a rat; let’s think a bit about it.” And that is absolutely fantastic.
By the way, the original question that I was asking to ChatGPT could look as it was intentionally trying to lead AI to a brute comparison of the databases. However, I also tried, in a different conversation, to ask “What should you answer to your client?” instead, and the response was pretty much the same. Google Bard haven't had any more luck either, falling into the same error of a comparison.