Subcases in ut cannot replace fixtures, and nor do mutable captures

[zhihaoy]

Problem

I switched from doctest to ut and have been using it extensively. So far, this is my biggest complaint. Doctest mostly eliminated the need for fixtures by introducing the tree-traversal interpretation of subcases (steps in ut). Consider

TEST_CASE("mut") {
    int i = 0;
    REQUIRE(i == 0);

    SUBCASE("return increased number for ++") {
        CHECK(++i == 1);
    }
    SUBCASE("return decreased number for --") {
        CHECK(--i == -1);
    }
}

This works naturally in doctest (Read this for details) because the execution of subcases is not sequential. Each subcase is deemed a node on a tree. To finish executing a test case, the control flow will traverse repeatedly all the way starting from the beginning to reach each leaf. Such an approach, by construction (no pun intended), isolates changes made to any variables to be declared outside of a subcase, treating all of them as established states, in other words, fixtures.

In ut, I would expect

"mut"_test = [] {
    int i = 0;
    expect((i == 0_i) >> fatal);

    should("return increased number for ++") = [] {
        expect(++i == 1_i);
    };
    should("return decreased number for --") = [] {
        expect(--i == -1_i);
    };
};

Of course, this doesn't compile. But the following would not work as (I) expected either at runtime:

"mut"_test = [] {
    int i = 0;
    expect((i == 0_i) >> fatal);

    should("return increased number for ++") = [&] {
        expect(++i == 1_i);
    };
    should("return decreased number for --") = [&] {
        expect(--i == -1_i);
    };
};

ut suggested mut or mutable captures:

"mut"_test = [] {
    int i = 0;
    expect((i == 0_i) >> fatal);

    should("return increased number for ++") = [=] {
        expect(++mut(i) == 1_i);
    };
    should("return decreased number for --") = [=] mutable {
        expect(--i == -1_i);
    };
};

But they come with drawbacks:

1. The variables to test must be copyable; move-only types cannot be "shared" in multiple subcases;
2. There is no guarantee that copies are distinct. shared_ptr is an obvious exception (but can appear quite frequently in some code bases). To put it in another way, a test's legitimacy now depends on a type's value semantics, but value semantics might be the thing I want to write the test for;
3. The usage of captures, & vs. =, affects the semantics of unrelated subcases. When obtaining a copy, the users must ensure it has stayed the same in previously declared subcases. The users cannot freely reorder subcases as a result;
4. When debugging, stepping inside a subcase means so many things have happened. In contrast, under the tree-traversal interpretation, stepping anywhere in the test case means only one branch has reached here.

Suggestion

Provide test steps (given, when, should, etc.) with tree-traversal interpretation in a separate namespace, maybe ut::isolation, as alternatives to existing test steps.