Home Home

Unit tests, stubs and non-deterministic behavior

Arseni Mourzenko
Founder and lead developer, specializing in developer productivity and code quality
130
articles
March 18, 2015

When it comes to unit testing, beginner programmers are often blocked when reaching code which feels non-deterministic or when the result changes over time.

Two classical examples are code which relies on pseudo-random numbers or on date and time.

In both cases, stubs are the solution. While dependency injection may make the code slightly more complicated, this is a low cost for a huge benefit of complete testability.

When recently refactoring a PHP project which had practically no tests, I encountered both of those classical cases for system tests. Every response of the API contained a random number (needed by XSLT to do some stuff) and many contained date-time entries which had every chance to change on every request.

So I added a specific mode which made it possible to stub both the pseudo-random number generator and the time generation function. As a result, I got perfectly reproducible responses from the API. System tests were based on those results, and ensured that no regression will be introduced during refactoring.

A similar case presented when I was rewriting an application from C# to Java. This application keeps the passwords of the websites, so I don't have to remember them nor use one password for every site. One part of this application is the generation of new passwords, with this method:

Warning: the code below contains a bug which will be explained later in the article.

public Password Generate(
        final int minLength,
        final int maxLength,
        final EnumSet<CharacterTypes> useTypes) {
    final String characters = useTypes
            .stream()
            .map(c -> c.getCharacters())
            .collect(Collectors.joining());

    final SecureRandom random = new SecureRandom();

    final Integer actualLength = maxLength == minLength ?
            minLength :
            random.nextInt(maxLength - minLength) + minLength;

    final String password = random
            .ints(actualLength, 0, characters.length())
            .boxed()
            .map(i -> characters.charAt(i))
            .map(c -> String.valueOf(c))
            .collect(Collectors.joining());

    return new Password(password);
}

The method generates a random password of a given length and containing specific characters, for instance CharacterTypes.CapitalLetter, CharacterTypes.SmallLetter, CharacterTypes.Digit. It concatenates the allowed characters in characters variable (for instance its value can be "ABCDEF…Zabcdef…z0123…9") and then randomly picks the characters from the string, repeating the operation n times, where n is in (minLength..maxLength) range.

The problem with this method is that as is, it cannot be tested. I can test a few things, such as:

  • That it returns 16 characters where both minLength and maxLength are equal to 16,

  • And that it uses only characters specified in useTypes parameter,

but both tests are not particularly useful (notice that both tests will result in code coverage close to 100%, which also shows how irrelevant code coverage can be in many cases). What would be useful is to test the logic itself, and to do it, the result should become deterministic.

First stub

The problem with the method is that it initializes SecureRandom. What if I don't need this sort of security, and just want to generate non-secure pseudo-random block of text?

By using Dependency injection, this initialization can be thrown out of the method, and delegated to the caller, which leads to the following code:

public class PasswordGenerator {
    private final Random randomProvider;

    public PasswordGenerator() {
        this(new SecureRandom());
    }

    public PasswordGenerator(final Random randomProvider) {
        this.randomProvider = randomProvider;
    }

    public Password Generate(...) {
        // Use `this.randomProvider` instead of `random`.
        ...
    }
}

This makes it possible to write our first stub:

private class RandomZeroStub extends Random {
    private static final long serialVersionUID = 1L;

    @Override
    public IntStream ints(
            final long streamSize,
            final int randomNumberOrigin,
            final int randomNumberBound) {
        return IntStream.iterate(0, i -> i).limit(streamSize);
    }

    @Override
    public int nextInt(final int bound) {
        return 0;
    }
}

This is a rather strange interpretation of randomness, but the deterministic behavior is exactly what I need for the tests. By knowing what not-so-random value is returned by the stub, I can test a few more things in Generate method, and more precisely:

  • That the minimum length criteria is respected, in other words that the password length will always be equal or superior to the value of minLength parameter. I couldn't test it before, because if there was a bug in my code, the test could sometimes pass, and sometimes fail.

  • That the actual password contains the characters of types specified in useTypes parameter. For instance, if I ask to generate the 8-characters password containing only capital letters, the only acceptable response using the stub would be “AAAAAAAA”.

Second stub

While the first stub got us pretty far, the code is still not tested very well, leaving a lot of room for bugs and regressions.

This is why a second stub is needed here. It is very similar to the first one in a sense that it too produces a deterministic output by always giving the same value, but unlike the first stub, this one gives the last value within a range.

private class RandomUpperBoundStub extends Random {
    private static final long serialVersionUID = 1L;

    @Override
    public IntStream ints(
            final long streamSize,
            final int randomNumberOrigin,
            final int randomNumberBound) {
        return IntStream.iterate(randomNumberBound - 1, i -> i).limit(streamSize);
    }

    @Override
    public int nextInt(final int bound) {
        return bound - 1;
    }
}

One should be cautious when implementing this stub: if we read the official documentation, it appears that the implementations of Random produce:

pseudorandom int values, each conforming to the given origin (inclusive) and bound (exclusive).

Noticed the exclusive keyword? That's why our RandomUpperBoundStub has - 1 in two locations.

This second stub makes it possible to write two additional tests:

  • The one which ensures that the length of the password is always equal or inferior to the value of maxLength,

  • And the one which ensures that the whole range of characters is used. The suspicion here is that it might be that when requesting a password containing capital letters, it will generate one containing only characters A to Y, but not Z.

While the second test passes, the first one fails, producing a password of length 7 for maxLength of 8. Great, we found a bug! The line which computes the length of the actual password should be replaced by:

final Integer actualLength = maxLength == minLength ?
    minLength :
    this.randomProvider.nextInt((maxLength - minLength) + 1) + minLength;

Noticed the + 1?

Conclusion

While numerous books explain that all business code should be covered by unit tests, few of them can give a clear illustration of the difficulties which can arise, and the corresponding solutions.

This leads numerous programmers to think that they shouldn't and cannot test parts of the code which rely on non-deterministic functions or on functions which have different output depending on the moment they are called. This leads to the most critical parts of the code being untested and prone to bugs and regressions.

While there are cases where testing is complicated, code relying on random numbers or time is not one of them. And code which actually cannot be tested represents far less than 0.1% of most code bases (actually, it is equal to 0% for many code bases, at least when it comes to business applications).

  • “I don't want to test this” may be a perfectly valid argument not to test.

  • “I can't test this” is usually not.