Unit Tests

A note on decomposition

Let's start from a side note on OOP. Basically, class methods are divided into two groups:

1. Getters — they return some values, but don't modify class.

auto get(/* args */) const;

2. Mutators — they return nothing, but modify class

void mutate(/* args */);

For example, let's look at std::queue:

// getters
bool empty() const;
size_t size() const;
const value_type& front() const;
const value_type& back() const;

// mutators
void pop();
void push(const value_type&);
void emplace(/* arguments */);

However, a lot of methods are 'mixed', due to bad design or complexity overhead

3A rule

Usually unit test writers follow the 3A rule:

1. Arrangement: prepare everything you need to start testing. The code you are using for arrangement should have been tested already, so you have nothing more to add.
2. Act: Now, do the action you want to test.
3. Assert: Receive the outcome and check it matches our expectations.

Catch2 library

Arrangement and act stages are performed in terms of common C++ code. The key point of testing is assertion, and it has to be easily readable, writeable, and debuggable: develop should immediately understand what test is failed, and proceed with analysis why it is happened.

There are many C++ frameworks which provide convinient environment for assertion purposes, like GoogleTest, Boost.Test, etc. We are using Catch2: it is portable, easy to start, and goodly supported. To use it, you have to a include just a single header!

#include <catch.hpp>

As many other frameworks, Catch2 combines single assertions into test cases: a single function intended to have a single arrangement, act, and assertion phases:

TEST_CASE("Testing arithmetics")
{
    // Arrange
    int a = 2;
    int b = 2;

    // Act
    int result = mul(a, b);

    // Assert
    CHECK(result == 4);
    CHECK_FALSE(result == 5);
}

As you may see, there are no strict rules for phases breakdown, because violating the 3A rule may be preferrable sometimes &emdash; but do not misuse that feature.

Test cases are intended to run separately and independently from each other. Ideally, you may run them in any order or in parallel, so avoid using static objects. The exception are static constants: it is much prettier to use some constant instead of numbers in your code:

TEST_CASE("Bad test case")
{
    Memory mem(65536);
    CHECK(mem.address_size() == 16);
}

static const size_t memory_size = 64 * 1024;
static const size_t address_size = 16; // to address 64 KB

TEST_CASE("Better test case")
{
    Memory mem(memory_size);
    CHECK(mem.size() == address_size);
}

Testing getters

If you want to test only getters, the first and second stages are combined. Let's take a look on example of MIPSRegister class:

TEST_CASE( "MIPS_registers: Hi_Lo_impossible")
{
    // Arranging: create 32 different MIPS registers
    MIPSRegister regs[32];
    for ( size_t i = 0; i < 32; ++i)
    {
        regs[i] = MIPSRegister(i);
    }

    // Asserting: expect that no register is HI or LO
    for ( const auto& reg : regs) {
        CHECK_FALSE(reg.is_mips_hi());
        CHECK_FALSE(reg.is_mips_lo());
    }
}

TEST_CASE( "MIPS_registers: Zero")
{
    // Arranging: creating a zero register
    auto reg = MIPSRegister::zero;

    // Asserting: checking all properties
    CHECK(reg.is_zero());
    CHECK_FALSE(reg.is_mips_hi());
    CHECK_FALSE(reg.is_mips_lo());
    CHECK(reg.to_size_t() == 0);
}

Testing mutators

How to test mutators? It will be easy if we follow 3A and have tests for getters already. The idea is to use getters to ensure that mutation happened correctly. For example, we are testing std::queue:

TEST_CASE("Queue: empty queue")
{
    // Arrangment: create a default queue
    std::queue<int> q;

    // Assert: check getters
    CHECK(queue.empty());
    CHECK(queue.size() == 0);
}

TEST_CASE("Queue: push value")
{
    // Arrangment: create a default queue
    // We are sure that default queue behaves like an empty queue
    std::queue<int> q;

    // Act: push some value
    q.push(2);

    // Assert
    CHECK(queue.front() == 2); // We expect to see 2 at front of the queue
    CHECK(queue.back() == 2); // We expect to see the same value at back
    CHECK_FALSE(queue.empty());
    CHECK(queue.size(), 1);
}

TEST_CASE("Queue: push two values")
{
    // Arrangment: create a queue and push a value
    // We are already sure that it behaves properly
    std::queue<int> q;
    q.push(2);

    // Act
    q.push(3)
    
    // Assert
    CHECK(queue.front() == 2); // We still expect to see 2 at front of the queue
    CHECK(queue.back() == 3); // We expect to see new value at back
    CHECK_FALSE(queue.empty());
    CHECK(queue.size(), 2);
}

TEST_CASE("Queue: push two values and pop")
{
    // Arrangment: create a queue and push two values
    // We are already sure that it behaves properly
    std::queue<int> q;
    q.push(2);
    q.push(3)

    // Act
    q.pop();

    // Assert
    CHECK(queue.front() == 3); // There should be 3 at front of the queue
    CHECK(queue.back() == 3); // We expect to see 3 at back
    CHECK_FALSE(queue.empty());
    CHECK(queue.size() == 1);
}

TEST_CASE("Queue: push two values and pop two values")
{
    // Arrangment: create a queue and push a value
    // We are already sure that it behaves properly
    std::queue<int> q;
    q.push(2);
    q.push(3)

    // Act
    q.pop();
    q.pop();

    CHECK(queue.empty()); // Must be empty now
    CHECK(queue.size() == 0); // Must have zero size
}

Testing internal methods of class

Basically, if you want to test an internal method of class means that your design is not optimal: you are testing internal methods because it is too hard to test external methods. Probably, the best solution is to extract your internal methods to a separate class.

However, if you work with legacy code, you may want to have tests before you start extracting new classes out of the old classes. The good example is RF class: it has internal std::array and accessing methods, then it has read and write methods which operate with MIPSRegister, and externally visible methods communicating with MIPSInstr. MIPS instruction class is already complicated, so our intention is to test RF without it.

How is it possible? There are two ways:

1. Hacker way. Never do this at home.

#define private public
#include "../rf.h"
#undef private

2. More safe and elegant way is to move methods you want to test to protected scope and define a following class in unit_test.cpp:

class TestRF : public RF {
public:
    // ctors

    // list of methods you want to use from basic class
    using RF::read;
    using RF::write;
};

Then, you may safely proceed with extraction of read/write methods to a separate class, like BasicRF.

Measuring code coverage

There are many tools to measure code coverage: percentage of code covered by tests. We are using CodeCov. It is free for open source projects, while it has a nice interface and integration to GitHub.

For each pull request, CodeCov runs unit tests and finds if there is a code which was never executed. The results are provided by CodeCov bot in a comment section of a pull request.