Unit tests

Most programmers have heard of unit tests. Many programmers regularly write unit tests, even though it should be all of them. Some even work with test-driven development, and this should be all of them, too. So, what are unit tests? Unit tests are the safety net which protects you from fixing critical bugs in the night after the release. They are your code’s lawyers and prove its innocence. Unit tests form the shiny armor which gives you courage before the battle of refactoring. Unit tests are the pillow which lets you find an untroubled sleep. In short, unit tests are invaluable tools for software development. This article presents some strategies how unit tests can be implemented in C++.

I still have no idea what a unit test is

Before we come to that, I still have to give you a useful answer to the question I posed right before I got carried away a bit. So, what are unit tests?

A unit test is a small piece of software, typically a function, which uses an isolated unit (function, class) of your production code in a certain pattern and asserts that it behaves as expected. It is important that the function is short, runs quickly, and produces an unambiguous result, i.e., “test passed” or “test failed”. No manual interaction is permissable, such as entering something on the command line or comparing things printed on the terminal. Unit tests should be written to be executed by a computer, and afterwards the computer should know if all tests passed or some tests failed.

Testing frameworks

Testing frameworks help with achieving this task. Do not reinvent the wheel here. Use one of the freely available unit testing frameworks, even though all of them have their quirks. It seems that CppUnit and googletest are the most popular ones currently. Microsoft’s unit test framework which comes with Visual Studio finds wide usage in the Windows world. The advantages of using such a framework are numerous:

• They provide functionality for registering tests locally. There is no need to modify a master file to include a new test function in your list of tests.
• They provide helper functions for testing common cases, such as comparing two values or checking that a certain exception is thrown.
• They reduce boiler-plate code, even though there is still a lot left.
• They let you run single tests, a suite of tests, or all tests. If your test runner is set up appropriately, all this is possible without recompiling.
• They collect the results of your tests and display them in an unambiguous way. You will know whether all your tests passed or some of them failed. In the latter case, the frameworks also tell you which ones.
• They are able to present the results in a machine-readable way, e.g., in an XML format. A build server can be configured to build and run your test code periodically and interpret the XML output. You will be notified if some change in your code broke the tests.

Get accustomed to a framework instead of writing your own or, worse, writing tests without one. The code snippets in this article employ the CppUnit framework.

Move tested code to a library

Testing is important, but you do not write code just to have a reason to write tests. You want to use your code in an actual project. Developing the code in a test project and then pasting it to an application project is not the way to go. Instead, move the code you want to test (this should be almost all your code) to a library project. Build either a shared or a static library. A second project contains your test code including the test runner. This project links against your library.

Finally, create a third project which contains application code which cannot be unit tested. This should not be much! The final application must be tested by executing the application with various input parameters. For console-based programs, this should be easy to test with python scripts, for example.

The triumvirat of library, test, and application projects ensure that the code you test is always the code you use.

Comparing values

The most common type of unit tests does little more than comparing two values with each other. Typically, expected and actual values should be the same. For this scenario, CppUnit provides a simple helper macro:

void test_size() {
	int const expected = 42;
	int const actual = add_two_values(17, 25);
	CPPUNIT_ASSERT_EQUAL(expected, actual);
}

For the simple task of comparing two values for equality, CPPUNIT_ASSERT_EQUAL(expected, actual) should be preferred over CPPUNIT_ASSERT(expected == actual). The difference between both is that the former prints the values of both expected and actual if the assertion fails, while the latter merely prints expected == actual. If you have a bug in your code, the extra information might come in handy.

Comparing floating points

Comparing floating point values is a little trickier due to the limited precision of such numbers. Thus, two floats or doubles should be considered equal even if they differ by a small value. For such cases, CppUnit offers the macro CPPUNIT_ASSERT_DOUBLES_EQUAL:

void approximate_pi()
{
	double const expected = 3.14;
	double const actual   = 22.0 / 7.0;
	double const tolerance = 1e-10;
	CPPUNIT_ASSERT_DOUBLES_EQUAL(expected, actual, tolerance);
}

Appropriate values for testing

Consider the following function which is supposed to calculate the volume of a three-dimensional body spanned by three vectors via the triple product:

double volume_of_body(vector3 const & v1, vector3 const & v2, vector3 const & v3)
{
	auto const base_area_normal = v2.cross_product(v3);
	auto const signed_volume = v1.scalar_product(base_area_normal);
	return std::abs(signed_volume);
}

When I test a function, I consider extreme input parameters first. They are often easy to identify or well-documented in the literature. In the above example, any vector could be zero, and thus the volume must be zero as well:

double test_empty_volume() 
{
	vector3 const v1 = {0, 0, 0};
	vector3 const v2 = {2.0, 3.0, 4.0};
	vector3 const v3 = {7.0, 6.0, 5.0};
	CPPUNIT_ASSERT_DOUBLES_EQUAL(0.0, volume_of_body(v1, v2, v3), 1e-12);
}

I would also write tests for the other vectors being zero. There are other boundary cases as well: Vectors could be parallel to each other or all vectors could span the same plane. In these cases, the volume would be zero as well. I would write a test to make sure.

Then, I choose values for which it is easy to see what should be the result even if it is not an extreme case:

double test_cube_volume() 
{
	vector3 const v1 = {0, 0, 2.0};
	vector3 const v2 = {2.0, 0, 0};
	vector3 const v3 = {0, 2.0, 0};
	CPPUNIT_ASSERT_DOUBLES_EQUAL(8.0, volume_of_body(v1, v2, v3), 1e-12);	
}

I would also write a test which ensures a positive volume even if the vectors are in the wrong (left-handed for all math freaks out there) order. Finally, an example with semi-random values completes the test. Use pen and paper to calculate the expected result.

Exceptions

After checking valid values one should always consider if there are invalid parameter ranges as well. It is highly recommended to express valid parameters as types as Markus explained in another article. Still, these specialized types have constructors which need to make sure that the type only holds values which satisfy a certain invariant. Make sure an exception is thrown if this is not the case. The following sample illustrates the concept:

class non_negative_double {
public:
	non_negative_double(double target_value) :
		value_(target_value)
	{
		if (target_value < 0.0) {
			throw std::invalid_argument("Value must be non-negative");
		}
	}

	// other methods
private:
	double value_;
};

A matching test for invalid values would be:

void test_invalid_value()
{
	CPPUNIT_ASSERT_THROW( non_negative_double invalid(-1e-12), std::invalid_argument);
}

Class hierarchies

Sometimes classes should be part of a class hierarchy. Consider the case that you write a custom exception class. The standard recommends that all exceptions are derived from std::exception, and many programmers utilize generic catch(std::exception const &) statements for exception handling. Thus, you should make sure your custom exception implements the std::exception interface with a test like this:

void implements_exception_interface()
{
	std::string const message("error message");
	custom_exception const error(message);
	std::exception const & error_via_base = error;
	CPPUNIT_ASSERT_EQUAL(message, std::string(error_via_base.what()));
}

To ensure that your class implements a certain interface, taking a base class reference to your object should be sufficient. It is better still to use the base class interface (what()) to assert the behavior of your child class. With a modern compiler, you may also use the metafunction std::is_base_of<Base, Derived>::value to find out if Derived is really derived from Base.

Constant member functions

Methods of a class are declared const when they guarantee to leave the logical state of the class alone (Actually, in the new standard const means that the standard library can safely assume your method is thread safe, i.e., it is possible to execute this method on the same object from concurrent threads). Check for the constness of a member function like this:

void size_is_const()
{
	custom_container container;
	container.insert(4);
	container.insert(5);
	custom_container const & const_container = container;
	CPPUNIT_ASSERT_EQUAL(std::size_t(2), const_container.size());
}

This test only compiles if custom_container‘s size() member function is declared const.

Size matters

A test has fulfilled its purpose not just because it passes. A test is supposed to tell the beautiful story of how a class or a function is intended to be used. Remember those lengthy novels back in school? Nobody likes long tests. Strive for short tests. Do not test too much in a single function, focus on a particular aspect. Have a look at the following example:

void test()
{
	custom_container container;
	container.insert(4);
	container.insert(5);
	container.insert(6);
	CPPUNIT_ASSERT_EQUAL(4, container.at(0));
	CPPUNIT_ASSERT_EQUAL(5, container.at(1));
	CPPUNIT_ASSERT_EQUAL(6, container.at(2));
	CPPUNIT_ASSERT_THROW(container.at(4), std::out_of_range);
	CPPUNIT_ASSERT_EQUAL(std::size_t(3), container.size());

	container.sort_descending();
	CPPUNIT_ASSERT_EQUAL(6, container.at(0));
	CPPUNIT_ASSERT_EQUAL(5, container.at(1));
	CPPUNIT_ASSERT_EQUAL(4, container.at(2));
	CPPUNIT_ASSERT_EQUAL(std::size_t(3), container.size());

	container.clear();
	CPPUNIT_ASSERT_EQUAL(std::size_t(3), container.size());
}

This test lacks focus, and because it does so it is rather lengthy. Above monster test could be broken into smaller tests:

void insert()
{
	custom_container container;
	container.insert(4);
	container.insert(5);
	CPPUNIT_ASSERT_EQUAL(4, container.at(0));
	CPPUNIT_ASSERT_EQUAL(5, container.at(1));
	CPPUNIT_ASSERT_EQUAL(std::size_t(2), container.size());
}

void at_invalid_index()
{
	custom_container container;
	CPPUNIT_ASSERT_THROW(container.at(0), std::out_of_range);
}

void sort()
{
	custom_container container;
	container.insert(4);
	container.insert(5);

	container.sort_descending();
	CPPUNIT_ASSERT_EQUAL(5, container.at(0));
	CPPUNIT_ASSERT_EQUAL(4, container.at(1));
}

void clear()
{
	custom_container container;
	container.insert(4);
	container.insert(5);

	container.clear();
	CPPUNIT_ASSERT_EQUAL(std::size_t(0), container.size();
	CPPUNIT_ASSERT_THROW(container.at(0), std::out_of_range);
}

The tests contain some duplication like filling the container with initial values. At the same time, each test is easily readable and illustrates what happens to the container when certain methods are called. Nevertheless, by adhering to the single responsibility principle we get better code.

Don’t repeat yourself

Don’t repeat yourself is a clean code principle which should be held high at all times. However, sometimes it interferes with other principles such as keep it simple, stupid. Does your test code feel bulky and repetitive? (Imagine the last sentence spoken with a TV add voice…) Try to identify commonalities and extract them to separate helper functions. For example, the following test function might further shorten above tests:

custom_container make_test_container()
{
	custom_container container;
	container.insert(4);
	container.insert(5);
	return std::move(container);
}

Factories for generic test objects are quite common. Other examples include small helper structs which contain variables often used together, for example mock facilities.

Avoid setup and teardown methods

Most frameworks offer test suites with setup() and teardown() methods. These methods are executed before and after each test in the suite. If you really want to use these methods, you are forced to make the objects you want to work with members of your test suite class. My advice is simple: do not use setup() and teardown() methods for C++ unit tests. Construct objects in your test functions as you need them. One size, i.e., setup(), rarely fits all tests. Your tests lose expressiveness and—if you do not pay attention—your tests interact with each other via member variables. The results of your tests might depend on the order in which they are executed. Some programmers even use pointers to be able to free currently unused members at will. There is no need to deal with evil. Construct objects locally and you will be fine.

As an afterthought, CppUnit’s setup() and teardown() may have been implemented blindly because these methods are actually useful in the Java world. For C++, the only situation where I can imagine these functions to be useful is when trying to test code which depends on the state of a singleton. Then again, singletons are a neverending source of pain and should be avoided! If your find yourself whishing for either setup() or teardown() in the C++ world, think about a better design for your production or test code.

Custom asserts

An even more important example of don’t repeat yourself is to write custom assert functions when a simple CPPUNIT_ASSERT is not enough. Consider the following example:

void stream_inserter()
{
	auto container = make_test_container();

	std::string const expected_output("[ 2: 4, 5 ]");
	std::ostringstream actual_output;
	actual_output << container;
	CPPUNIT_ASSERT_EQUAL(expected_output, actual_output.str());

	container.insert(6);
	std::string const expected_output2("[ 3: 4, 5, 6 ]");
	std::ostringstream actual_output2;
	actual_output2 << container;
	CPPUNIT_ASSERT_EQUAL(expected_output2, actual_output2.str());
}

Checking the result of the << operator is technical and little more than overhead in the test code. The duplication should make that apparent. Let us clean up the code by introducing a little helper function:

void assert_stream_output(std::string const & expected_output, custom_container const & container)
{
	std::ostringstream actual_output;
	actual_output << container;
	CPPUNIT_ASSERT_EQUAL(expected_output, actual_output.str());
}

void stream_inserter()
{
	auto container = make_test_container();

	assert_stream_output("[ 2: 4, 5 ]", container);
	container.insert(6);
	assert_stream_output("[ 3: 4, 5, 6 ]", container);
}

With our custom assert_stream_output() function the intent of this test is obvious.

A word on MACROS

As I already mentioned a few times, I do not like macros because I do not trust them. CppUnit provides a number of helper macros. They make life, i.e., syntax (don’t tell my therapist), a little easier. The price I pay is barely comprehensible error messages and a few workarounds I have to use when trying to test template code. Try to use CppUnit without macros and see for yourself what you prefer. All in all, I think using those pesky little beasts is better than writing your own framework.

There is no second-class code

People tend to think that writing tests is a duty that comes with the job and is best gotten over with quickly. I could not disagree more. As your code evolves, your tests must evolve, too. I am certain that most of us know a project (someone else’s, of course) in which every change is a nightmare, because the code is an intricate, fragile, and incomprehensible mess. If this is the state of the code, there might be some money in betting on the state of the tests (provided they exist at all).

A chef who does not care for his knives is a rare sight. As a professional or hobbyist programmer, expect test code to meet the same high standards as your production code. Your future self will thank you later for stable, maintainable tests. Your tests will also form a beautiful documentation of how your code is meant to be used.

There is no second class code. However, your test’s documentation aspect may require a different balance between keep it simple and don’t repeat yourself. Try different styles and find out what works best. It is worth the effort.