Evaluation of Catch2

[hobovsky]

I managed to install Catch2 locally (the amalgamated version) and run the example test suite. I will use this discussion as an area to dump potentially interesting facts I managed to figure out. I think that format of a discussion and its functionality of threads will help with maintaining the order.

[hobovsky]

Organization of tests

Great advantage of Catch2 is that is allows for arbitrary titles of sections.

I think I figured out why you could not get the reporter to behave nicely. It turns out that Catch2 uses quite peculiar way of organization of tests: it does not follow the CW's-favorite describe/it/assert way of organization. Instead, it seems to resemble Julia's FactCheck, where Catch2's TEST_CASE is FactCheck's @facts, SECTION seems to correspond to @context, and REQUIRE is a @fact. As a result, it can be difficult to make a reporter which handles the organization artifacts with <DESCRIBE::> and <IT::>, because we will encounter similar problems as with Julia's setup, where we do not know what is an outer level, and what is the innermost level. Catch2's BDD mode does not seem to be helpful here either, because the GIVEN/WHEN/THEN just map to SECTION and we still do not know when we are in the innermost one. It could work if we forced some CW-specific convention, but I think it's not something what we want. Or, if the reporter can keep/pass state between events, it can try to deduce if it needs to emit an <IT::> at the point it attempts to emit <PASSED::> or <FAILED::>. I can give it a shot, but this this also feels a bit off (related: codewars/runner#262).

[hobovsky]

See this kumite for a result of running a test suite with multiple nesting levels, where every group is a mixture of assertions and inner groups.

The code of the test suite which generated this report is present in the description of the kumite, but it can also be found in the ExampleOrganization example in my fork.

[hobovsky]

Assertion messages

It seems that Catch2 does not support assertion messages inside of its REQUIRE/CHECK macros. It is possible to provide custom messages with matchers, but depending on organization of tests, this can be cumbersome, especially in context of CW and the practice of random tests, if authors wanted to have an explicit feedback for a failing random case.

There is a feature request in Catch2's repository to provide such messages: catchorg/Catch2#2653 but it was already shot down once (catchorg/Catch2#929).

[hobovsky]

Reporter

The reporter is notoriously difficult to get to work for a couple of reasons:

• It is difficult to decide where to output an <IT::> marker (see Organization of tests above).
• Whenever a section starts, the framework re-reports all the events related to all sections above the tested one (i.e. when a section is started, all events related to ancestor sections and test cases are re-raised). This can be probably worked around by tracking the history of sections in the reporter (some example reporters already do this), but has to be evaluated further.
• Passed assertions are not reported by default. This might be not necessary, but can be helpful if assertions would be used to deduce whether an <IT::> should be emitted or not. To report passing assertions, the reporter needs to have m_preferences.shouldReportAllAssertions = true; in its constructor.
• The flag m_preferences.shouldRedirectStdOut = true;, should it be used, conflicts with the functionality of writing to stdout by the reporter. For now, the flag does not seem to be necessary and things written to stdout from inside of solution and from inside of tests appear in the report when the flag is set to false, but docs in some place suggested that the flag can be useful sometimes.

Still need to evaluate:

• reporting of generated test cases: nesting, failures and success,
• testing for (or for lack of) errors, exceptions, and failures,
• report in case of a fatal failure.

[hobovsky]

See this message for an attempt at filtering out potentially unnecessary events (related to implicit sections). Still, the Partial events are somewhat noisy and maybe could be skipped too, but then some information will be lost.

Innermost group are synthetic <IT::> generated around every assertion, like in Julia reporter. It does not look pretty, but works.

[hobovsky]

Build

Catch2, as opposed to header-only libraries like for example Snowhouse, is a library which needs to be linked with tests:

• it can be either pre-built and embedded in the runner image to be referenced by tests, or
• it can be rebuilt and linked on every run of tests, or
• its amalgamated version can be embedded into the source tree of a CW kata, and build together with the kata.

Locally i work with the amalgamated setup embedded in my local project, but I do not know which of three above options will be the best for CW runner. I expect prebuilding the library would save some time for user solutions.