Collection contexts

[stony8]

In attest, tests may have contexts. What about contexts for test
collections? They would be useful for expensive setup/teardown of a
collection's "environment".

I know that they would not be trivial to add since, currently, Tests.register()
merges/"flattens" in tests of a "sub-collection".

Have collection contexts been considered? If yes, have they been left out as a
feature decision/for simplicity?

[dag]

Mainly left out so far because:

• I'd like test functions to work correctly if called directly
• Proper testing should ensure isolation I think

But the latter is not a strong enough reason for me to leave it out; if you know what you're doing, collection-wide contexts can be useful as you say. The former is more tricky, as you also noted. Would be easy to make it work per function, just treat collection contexts as test contexts, but for running a whole collection something different would need to happen.

I think it might be somewhat trivial to add shared contexts for everything in a collection but I think it should not be for "sub-collections". A collection would need to be defined with more assumptions than "iterable yielding tests".

[dag]

Maybe we could have 'shared contexts' that keep track of the tests they surround, and only runs the setup the first time we 'enter', and the teardown when all tracked tests have 'exited'. Unless I'm confused, I think that could work while keeping tests-as-simple-callables and collections-as-simple-iterables.

[dag]

…except it wont work for running individual tests by calling, if not all tests in the collection are run. The teardown would never run. Exit handlers could be used to ensure teardown is handled before the process exits, which will not work optimally if the process is kept alive.

Running individual tests and keeping processes alive are probably rare uses, so it's probably "good enough", with the gotcha's documented. Some way to trigger all teardown could also be added, maybe with some use of signals and a context manager for enclosing a run of arbitrary tests.