Comparative Review of FLOSS Testing Frameworks for Embedded C++

Requirements

It turns out that most automated testing frameworks for C++ were designed for Desktop, as opposed to Embedded devices. They usually report test results to standard output, many frameworks even supply main() function. On Embedded platforms we usually don’t have stdout and applications can not use main() function, generated by the frameworks. We need to capture testing report into memory and then either output it to the log, or to some nice window on the device.

Thus, our requirements for testing frameworks are as follows:

• Cross-platform, i.e. support for testing on Android, iOS, Linux, MacOS X, Windows.
• Support for custom test runner, because we don’t have main() on Android or iOS.
• Support for custom outputter, because we don’t have stdout on Android and iOS.
• Support for fixtures, i.e. setup/teardown.
• Support for testsuites, i.e. test grouping.

Non-mandatory, but desired features:

• Easy and pleasant to use: Sensible and logical design, good documentation, sensible syntax, terminology and keywords.
• The framework should be supported and mature.
• The less boilerplating – the better. For example, automatic test registration.
• Support for running test selectively, preferably by regex match on test names.
• Support for listing available tests.

Testing framework list

The following testing frameworks were evaluated:

• Bandit
• Boost.Test
• CATCH
• CppUnit
• CxxTest
• Google Test
• Igloo
• Lest
• TUT
• UnitTest++

More about each framework

CppUnit

C++’s classics of the classics. The first xUnit-like testing framework for C++. Powerful and feature-rich. Has good documentation. Unfortunately, test registration is not automatic and developer will need to retype test class and function names several times. xUnit-like frameworks in other languages, like Java and Ruby, do not require such manual test registration, because they, unlike C++, have reflection mechanisms, which are used for automatic test discovery.

Code example:

class MyTestSuite : public CppUnit::TestFixture
{
public:
    void setUp()
    {
        m_num = 2;
    }

    void tearDown()
    {
        m_num = 0;
    }

    void testOneThing()
    {
        CPPUNIT_ASSERT(m_num == 2);
    }

    void testAnotherThing()
    {
        CPPUNIT_ASSERT_EQUAL(m_num * m_num, 4);
    }
    ...
};

...

auto* suite = new CppUnit::TestSuite("MyTestSuite");

suite->addTest(
    new CppUnit::TestCaller <MyTestSuite> (
        "testOneThing",
        &MyTestSuite::testOneThing
    )
);

suite->addTest(
    new CppUnit::TestCaller <MyTestSuite> (
        "testAnotherThing",
        &MyTestSuite::testAnotherThing
    )
);

CppUnit::TextUi::TestRunner runner;

runner.addTest(suite);

Supports:

• Custom runner.
• Custom outputter by subclassing class TextOutputter and overriding 1 function.
• Fixtures and testsuites.
• Listing available tests.
• Selective running of one particular test.
• Unlike most other frameworks, defining and registering test both using C++-only code and using macros.

Downsides:

• Test autoregistration is almost non-existing. There are helper macros, but they do not help enough.
• This framework requires much boilerplating. People on the Internet complain about it a lot.
• The framework is supported, but not very actively. Currently it is supported by LibreOffice team. On the other hand, the framework already has so many features, that it does not need much further development. Sure, there is a room for improvement with boilerplating, but such work suggests so much refactoring that it is easier to just take another testing framework.

Verdict:

CppUnit requires too much boilerplating. It was probably OK 15 years ago, when C++ developers did not have much choice, but for 2016 it is too bad.

Google Test

Powerful framework with lots of features and complicated syntax. Has good documentation. Unfortunately, it is hard to redirect output. It is one of the first testing frameworks, featuring automatic test registration. C++ still does not have reflection, but Google Test overcomes this obstacle by using macros, that both define and register tests.

Supports:

• Custom runner.
• Fixtures and testsuites.
• Test autoregistration.
• Listing available tests.
• Running subset of tests, including and excluding them by path-like wildcards.

Downsides:

• Custom outputter is not supported. The framework uses C file descriptors for output. The best that can be done is redirecting output to the file.

Verdict:

Google Test is not good enough for us, because custom outputter is not supported.

Boost.Test

Powerful framework with lots of features and complicated syntax. Has good documentation. Did not support autoregistration before, but supports now. Can be compiled as static or dynamic library or used as header-only library. People on the Internet report that in case of header-only library compilation takes quite long time. Typical for a Boost library.

Code example:

struct MyFixtureStructure
{
    MyFixtureStructure()  { m_num = 2; }
    ~MyFixtureStructure() { m_num = 0; }
    ...
};

BOOST_FIXTURE_TEST_SUITE( MyTestSuite, MyFixtureStructure )

    BOOST_AUTO_TEST_CASE( test_one_thing )
    {
        BOOST_REQUIRE(m_num == 2);
    }

    BOOST_AUTO_TEST_CASE( test_another_thing )
    {
        BOOST_CHECK_EQUAL(m_num * m_num, 4);
    }

BOOST_AUTO_TEST_SUITE_END()

Supports:

• Custom runner.
• Custom outputter by subclassing std::ostream.
• Fixtures and testsuites.
• Test autoregistration.
• Listing available tests.
• Running subset of tests, selecting by path-like wildcards and tags.

Verdict:

Good candidate, supports all our requirements. Can we find better?

CxxTest

Lightweight framework with good design and syntax. Implements automatic testcase registration by running a Python script, instead of clumsy macros, used by other testing frameworks. Each testsuite is a class with (optional) setUp/tearDown functions, each test is a function starting with “test”. As a result, a testsuite looks like a nice C++ class. The framework has good, even though not very long, documentation and easily readable source code otherwise.

Code example:

class MyTestSuite : public CxxTest::TestSuite
{
public:
    void setUp()
    {
        m_num = 2;
    }

    void tearDown()
    {
        m_num = 0;
    }

    void testOneThing()
    {
        TS_ASSERT(m_num == 2);
    }

    void testAnotherThing()
    {
        TS_ASSERT_EQUALS(m_num * m_num, 4);
    }
    ...
};

Supports:

• Custom runner.
• Custom outputter by subclassing class OutputStream and overriding 3 functions.
• Fixtures and testsuites.
• Test autoregistration without macros.
• Listing available tests.
• Selective running of one particular test or testsuite.

Downsides:

• Introduces dependency on Python for a C++ project.
• The Python script, used for testing code processing, has simplified C++ parser. Thus testing code must be kept parser-friendly. Fortunately, it is not hard.

Verdict:

Good candidate, supports all our requirements. Avoids macros, code looks cleaner. Can we still find better?

CATCH

New-generation testing framework, supporting both usual xUnit-style tests and TDD/BDD-style tests with SECTIONS and SCENARIOS. SECTIONS is a killer feature, allowing to save a lot of code on fixtures. SCENARIOS is development of the SECTIONS idea, more descriptive, but requires more typing.

Code example:

TEST_CASE( "My test suite name", "[my_tag]" )
{
    num = 2;

    SECTION( "increment" )
    {
        num++;
        REQUIRE(num == 3);
    }

    SECTION( "decrement" )
    {
        num--;

        SECTION( "increment after decrement" )
        {
            num++;
            REQUIRE(num == 2);
        }

        SECTION( "2 decrements" )
        {
            num--;
            REQUIRE(num == 0);
        }
    }
}

Using SECTIONS, it is possible to combine fixture code with testing code. The above code is equivalent to 3 tests in xUnit model:

TEST_CASE( "increment" )
{
    num = 2;
    num++;
    REQUIRE(num == 3);
}

TEST_CASE( "increment after decrement" )
{
    num = 2;
    num--;
    num++;
    REQUIRE(num == 2);
}

TEST_CASE( "2 decrements" )
{
    num = 2;
    num--;
    num--;
    REQUIRE(num == 0);
}

As we can see, SECTIONS provide a way to compactly describe several tests in a tree-like manner.

Supports:

• Custom runner.
• Custom outputter by subclassing std::ostream.
• Fixtures and testsuites.
• SECTIONS and SCENARIOS!
• Test autoregistration.
• Listing available tests.
• Running subset of tests, selecting by path-like wildcards and tags.

Verdict:

Surprisingly good young contender. Supports all our requirements and in addition has SECTIONS killer feature. We have a winner!

Evaluation of other testing frameworks follows for completeness of the review.

Lest

Aims to be C++11-fied version of CATCH. Seems to be less powerful than CATCH so far, and less mature. The first commit on GitHub for lest was in June 2013, vs November 2010 for CATCH. According to lest homepage, lest takes much more time to compile, probably because of excessive use of C++11 features.

Igloo

BDD-style framework with unclear syntax and bad documentation. Seems to be inactively supported: the last commit on GitHub is from July 2015, but the last release was in 2013. Seems like the author devotes his attention to his another testing framework, Igloo.

Bandit

Another BDD-style framework with unclear syntax and bad documentation. C++11-fied version of Igloo framework from the same author. C++11 syntax was supposed to make the syntax better, but, I believe, the opposite happened: Bandit syntax is even worse than Igloo syntax, despite that the author calls it “Human friendly unit testing for C++11”.

TUT

Old framework with awful syntax, relying on C++ templates instead of C++ macros. Unsupported: the last release was in 2013, commit rate is approximately 2 commits per year.

UnitTest++

Minimalistic framework with “usual” syntax with C++ macros TEST/SUITE/CHECK/etc. Documentation is quite brief. The framework seems to be supported, though not much development is being done, probably because the intention is to keep the framework minimalistic. The last release and the last commit was in November 2015.

Supports:

• Custom runner.
• Custom outputter by subclassing class TestReporter and overriding 4 functions.
• Fixtures and testsuites.
• Test autoregistration.

Downsides:

• No support for listing available tests.
• No support for selective test running.

Verdict:

Good choice, if you want very light-weight and minimalistic framework. If you want more features – choose something else.

Conclusion

Considering upsides and downsides of different testing frameworks, I am giving top 3 places to these frameworks that satisfy all our requirements:

1. CATCH, for supporting SECTIONS.
2. CxxTest, for avoiding clumsy macros, easy implementation of custom outputter and generally good design.
3. Boost.Test, for being feature-rich, mature and quality product.