Evaluation of Google Test

[hobovsky]

I managed to build Google Test locally and run the example test suite. I will use this discussion as an area to dump potentially interesting facts I managed to figure out.

[hobovsky]

Organization of tests

A disadvantage of Google Test is that titles of sections have to be valid identifiers. It even has a whole paragraph on what is a valid identifier.

GoogleTest seems to use organization very similar to Igloo, with the test_suite name > test name scheme. What is interesting, The Visual Studio Test Adapter seems to group by namespace of a test class, and maybe such additional nesting would be useful also for us?

[hobovsky]

See ExampleOrganization in my fork for examples of various test groups and fixtures.

[hobovsky]

Assertion messages

GoogleTest supports additional assertion messages by EXPECT_EQ(condition) << message;

[hobovsky]

Examples of failure messages are presented by ExampleFailureMessages in my fork.

[hobovsky]

Reporter

The reporter seems to work well for:

• passing tests,
• failing tests,
• printing from inside of a solution,
• printing from inside a test.
• reporting of generated test cases (TEST_P): nesting, failures and success,
• testing for (or for lack of) exceptions (see ExampleExceptions),

When a SUT or a test crashes, test run is immediately interrupted. The report preceding the crash is produced normally, but it breaks abruptly whether a crash occurs in an assertion or not (see ExampleCrash).

[hobovsky]

TEST_P (i.e. parametrized tests) generate oh so ugly report (see ExampleParametrized). Maybe it could be processed a bit in the reporter, but in its default form it looks quite ugly. However, I think not many authors like parametrized tests on CW in general, so maybe it's not a big issue.

[hobovsky]

@kazk I am not sure what you mean by your TODO note in the reporter when printing an <IT::>:

// TODO Group by `test_case_name`? It's not identifiers so uniqueness is not guaranteed.

Tests are grouped by their test case (i.e. test suite) out of the box, so I am not sure what additional grouping would be needed? Additionally, removing the test_case_name from titles of tests improves the output significantly, and even parametrized tests look much better without it.

[kazk]

I have no idea, so you can ignore it 😆 Go ahead and remove test_case_name from the titles of tests.

[hobovsky]

Build

Google Test, as opposed to header-only libraries like for example Snowhouse, is a library which needs to be linked with tests:

• it can be either pre-built and embedded in the runner image to be referenced by tests (as a dynamic or a static library), or
• it can be rebuilt and linked on every run of tests (recommended by Google!).
• it can be installed from a package repository.

Locally i work with the prebuilt static library, but I do not know which of the above options will be the best for CW runner. I expect prebuilding the library would save some time for user solutions.

[hobovsky]

Note that ASSERT_THAT macro and matchers are not a part of Google Test, but rather of gMock. If Google Test is installed from a package repository, it might be necessary to add gMock explicitly.