Unit tests, stubs and non-deterministic behavior

When it comes to unit test­ing, be­gin­ner pro­gram­mers are of­ten blocked when reach­ing code which feels non-de­ter­min­is­tic or when the re­sult changes over time.

Two clas­si­cal ex­am­ples are code which re­lies on pseu­do-ran­dom num­bers or on date and time.

In both cas­es, stubs are the so­lu­tion. While de­pen­den­cy in­jec­tion may make the code slight­ly more com­pli­cat­ed, this is a low cost for a huge ben­e­fit of com­plete testa­bil­i­ty.

When re­cent­ly refac­tor­ing a PHP pro­ject which had prac­ti­cal­ly no tests, I en­coun­tered both of those clas­si­cal cas­es for sys­tem tests. Every re­sponse of the API con­tained a ran­dom num­ber (need­ed by XSLT to do some stuff) and many con­tained date-time en­tries which had every chance to change on every re­quest.

So I added a spe­cif­ic mode which made it pos­si­ble to stub both the pseu­do-ran­dom num­ber gen­er­a­tor and the time gen­er­a­tion func­tion. As a re­sult, I got per­fect­ly re­pro­ducible re­spons­es from the API. Sys­tem tests were based on those re­sults, and en­sured that no re­gres­sion will be in­tro­duced dur­ing refac­tor­ing.

A sim­i­lar case pre­sent­ed when I was rewrit­ing an ap­pli­ca­tion from C# to Java. This ap­pli­ca­tion keeps the pass­words of the web­sites, so I don't have to re­mem­ber them nor use one pass­word for every site. One part of this ap­pli­ca­tion is the gen­er­a­tion of new pass­words, with this method:

Warn­ing: the code be­low con­tains a bug which will be ex­plained lat­er in the ar­ti­cle.

public Password Generate(
        final int minLength,
        final int maxLength,
        final EnumSet<CharacterTypes> useTypes) {
    final String characters = useTypes
            .stream()
            .map(c -> c.getCharacters())
            .collect(Collectors.joining());

    final SecureRandom random = new SecureRandom();

    final Integer actualLength = maxLength == minLength ?
            minLength :
            random.nextInt(maxLength - minLength) + minLength;

    final String password = random
            .ints(actualLength, 0, characters.length())
            .boxed()
            .map(i -> characters.charAt(i))
            .map(c -> String.valueOf(c))
            .collect(Collectors.joining());

    return new Password(password);
}

The method gen­er­ates a ran­dom pass­word of a giv­en length and con­tain­ing spe­cif­ic char­ac­ters, for in­stance CharacterTypes.CapitalLetter, CharacterTypes.SmallLetter, CharacterTypes.Digit. It con­cate­nates the al­lowed char­ac­ters in characters vari­able (for in­stance its val­ue can be "ABCDEF…Zabcdef…z0123…9") and then ran­dom­ly picks the char­ac­ters from the string, re­peat­ing the op­er­a­tion n times, where n is in (minLength..maxLength) range.

The prob­lem with this method is that as is, it can­not be test­ed. I can test a few things, such as:

• That it re­turns 16 char­ac­ters where both minLength and maxLength are equal to 16,
• And that it uses only char­ac­ters spec­i­fied in useTypes pa­ra­me­ter,

but both tests are not par­tic­u­lar­ly use­ful (no­tice that both tests will re­sult in code cov­er­age close to 100%, which also shows how ir­rel­e­vant code cov­er­age can be in many cas­es). What would be use­ful is to test the log­ic it­self, and to do it, the re­sult should be­come de­ter­min­is­tic.

First stub

The prob­lem with the method is that it ini­tial­izes SecureRandom. What if I don't need this sort of se­cu­ri­ty, and just want to gen­er­ate non-se­cure pseu­do-ran­dom block of text?

By us­ing De­pen­den­cy in­jec­tion, this ini­tial­iza­tion can be thrown out of the method, and del­e­gat­ed to the caller, which leads to the fol­low­ing code:

public class PasswordGenerator {
    private final Random randomProvider;

    public PasswordGenerator() {
        this(new SecureRandom());
    }

    public PasswordGenerator(final Random randomProvider) {
        this.randomProvider = randomProvider;
    }

    public Password Generate(...) {
        // Use `this.randomProvider` instead of `random`.
        ...
    }
}

This makes it pos­si­ble to write our first stub:

private class RandomZeroStub extends Random {
    private static final long serialVersionUID = 1L;

    @Override
    public IntStream ints(
            final long streamSize,
            final int randomNumberOrigin,
            final int randomNumberBound) {
        return IntStream.iterate(0, i -> i).limit(streamSize);
    }

    @Override
    public int nextInt(final int bound) {
        return 0;
    }
}

This is a rather strange in­ter­pre­ta­tion of ran­dom­ness, but the de­ter­min­is­tic be­hav­ior is ex­act­ly what I need for the tests. By know­ing what not-so-ran­dom val­ue is re­turned by the stub, I can test a few more things in Generate method, and more pre­cise­ly:

• That the min­i­mum length cri­te­ria is re­spect­ed, in oth­er words that the pass­word length will al­ways be equal or su­pe­ri­or to the val­ue of minLength pa­ra­me­ter. I couldn't test it be­fore, be­cause if there was a bug in my code, the test could some­times pass, and some­times fail.
• That the ac­tu­al pass­word con­tains the char­ac­ters of types spec­i­fied in useTypes pa­ra­me­ter. For in­stance, if I ask to gen­er­ate the 8-char­ac­ters pass­word con­tain­ing only cap­i­tal let­ters, the only ac­cept­able re­sponse us­ing the stub would be “AAAAAAAA.”

Sec­ond stub

While the first stub got us pret­ty far, the code is still not test­ed very well, leav­ing a lot of room for bugs and re­gres­sions.

This is why a sec­ond stub is need­ed here. It is very sim­i­lar to the first one in a sense that it too pro­duces a de­ter­min­is­tic out­put by al­ways giv­ing the same val­ue, but un­like the first stub, this one gives the last val­ue with­in a range.

private class RandomUpperBoundStub extends Random {
    private static final long serialVersionUID = 1L;

    @Override
    public IntStream ints(
            final long streamSize,
            final int randomNumberOrigin,
            final int randomNumberBound) {
        return IntStream.iterate(randomNumberBound - 1, i -> i).limit(streamSize);
    }

    @Override
    public int nextInt(final int bound) {
        return bound - 1;
    }
}

One should be cau­tious when im­ple­ment­ing this stub: if we read the of­fi­cial doc­u­men­ta­tion, it ap­pears that the im­ple­men­ta­tions of Random pro­duce:

pseudo­ran­dom int val­ues, each con­form­ing to the giv­en ori­gin (in­clu­sive) and bound (ex­clu­sive).

No­ticed the ex­clu­sive key­word? That's why our RandomUpperBoundStub has - 1 in two lo­ca­tions.

This sec­ond stub makes it pos­si­ble to write two ad­di­tion­al tests:

• The one which en­sures that the length of the pass­word is al­ways equal or in­fe­ri­or to the val­ue of maxLength,
• And the one which en­sures that the whole range of char­ac­ters is used. The sus­pi­cion here is that it might be that when re­quest­ing a pass­word con­tain­ing cap­i­tal let­ters, it will gen­er­ate one con­tain­ing only char­ac­ters A to Y, but not Z.

While the sec­ond test pass­es, the first one fails, pro­duc­ing a pass­word of length 7 for maxLength of 8. Great, we found a bug! The line which com­putes the length of the ac­tu­al pass­word should be re­placed by:

final Integer actualLength = maxLength == minLength ?
    minLength :
    this.randomProvider.nextInt((maxLength - minLength) + 1) + minLength;

No­ticed the + 1?

Con­clu­sion

While nu­mer­ous books ex­plain that all busi­ness code should be cov­ered by unit tests, few of them can give a clear il­lus­tra­tion of the dif­fi­cul­ties which can arise, and the cor­re­spond­ing so­lu­tions.

This leads nu­mer­ous pro­gram­mers to think that they shouldn't and can­not test parts of the code which rely on non-de­ter­min­is­tic func­tions or on func­tions which have dif­fer­ent out­put de­pend­ing on the mo­ment they are called. This leads to the most crit­i­cal parts of the code be­ing untest­ed and prone to bugs and re­gres­sions.

While there are cas­es where test­ing is com­pli­cat­ed, code re­ly­ing on ran­dom num­bers or time is not one of them. And code which ac­tu­al­ly can­not be test­ed rep­re­sents far less than 0.1% of most code bases (ac­tu­al­ly, it is equal to 0% for many code bases, at least when it comes to busi­ness ap­pli­ca­tions).

• “I don't want to test this” may be a per­fect­ly valid ar­gu­ment not to test.
• “I can't test this” is usu­al­ly not.