Using Fixtures as Parameters

[jgersti]

In #11284 @RonnyPfannschmidt mentioned that he would like to incorporate pytest-lazy-fixture¹ and building blocks/hooks for pytest-cases² like behaviour into the pytest core.
I have experimented the last month with a pytest-cases² replacement since its author has not been active for some time.
I would like to share some my observations and conclusion to start a discussion about what the scope of the feature should be and what steps need to be taken to implement it.

I will start by giving a brief overview what these plugins do and how they work, followed by my opinion what is in scope for addition to the core of pytest. In case of pytest-cases this description is heavily simplified.

Plugin Overview

pytest-lazy-fixture¹

The plugin provides a single feature: it introduces a LazyFixture object to reference a fixture inside pytest.mark.parametrize and the params argument of pytest.fixture.

Because the referenced fixture maybe parametrized itself or have a parametrized dependency, a pytest_generate_tests hook is used after the core test generation to discover transitive parameters by inspecting each Metafunc.callspecs funcargs and params entries for LazyFixture objects and recursively descending until no further parameters are found.
This is done by recalculating a new fixture closure with the referenced fixture included and calling FixtureManager.pytest_generate_tests with a new (deep) copied Metafunc object with the new fixture closure and finally replacing the callspecs in the old Metafunc are replaced by the newly calculated ones when ascending.
Because the added fixture names in the new fixture closure cannot be passed to the Metafunc object without influencing all calls, they are dropped at this point and in turn are not considered when reordering the tests and higher scoped fixture maybe initialised late and/or multiple times. Also since the additional parametrizations are applied last instead of in order of the dependencies the parameter id order is wrong.

The LazyFixture objects are resolved by using the pytest_run_setup, pytest_fixture_setup and pytest_run_call hooks.
In pytest_runtest_steup item._request._fillfixtures is replaced with a wrapper that inspects item.callspec.params and item.funcargs and resolves found LazyFixtures before calling the original _fillfixture method.
During iteration item.callspec.params is reordered to account for dependency order and scopes.
In pytest_fixture_setup request.param is inspected and if it is a LazyFixture resolved, resp. in pytest_runtest_call each item.funcargs entry is inspected and resolved.

pytest-cases²

The plugin provides an (opiniated³) unified alternative to conventional parametrization. It features a two new decorators for parametrization, parametrize and parametrize_with_cases, a replacement decorator for pytest.fixture and lazily evaluated functions as parameters and fixture reference in form of FixtureRef (parametrize can auto detect fixtures and wrap them into a FixtureRef).
The plugin also provides some more features but to keep the description short I will concentrate on the mentioned featured minus lazy functions I will also not delve into how cases are discovered and just assume we are given a list of functions as cases.

The basic idea is offload parametrization to the new parametrize decorator and use it for test functions and fixtures (as well as cases) to have a unified UX.
This is achieved by wrapping the function that is decorated to manipulate the signature, creating an intermediate fixture that is parametrized and creating a parameter fixture for each parameter (if the parameter is a FixtureRef this parameter fixture is obviously skipped and the referenced fixture is used).
The intermediate fixture depends on all parameter fixtures but during test collection and execution fixtures that are created using the new fixture decorator are selectively disabled.
Indirect parametrization is not allowed if any new feature is used.
The new fixture decorator does not support the params argument but detects parametrization marks and create the parametrization similar to the above mechanism.
It also wraps the decorated function to inject the mechanism for skipping.
Case parametrization decorator takes a list of function (or a class) and applies the fixture decorator to it and forwards FixtureRefs for these to parametrize.

To achieve the proper parametrized test and proper skipping of all the parameters that are not currently active, the plugin replaces FixtureManager.getfixtureclosure and wraps Metafunc.parametrize to inject facades into FuncFixtureInfo.names_closure and Metafunc.callspec.

Note: This is a extremely simplified description that hopefully conveys the major points.

Feature Scope

While a unified parametrization UX would be nice to have it is most definitely out of scope because it would break most existing code bases and would involve quite a bit of black magic behind the curtains.

What I think is in scope is a reimplementation of pytest-lazy-fixture, though I would prefer a name like FixtureRef/FixtureReference because it better conveys the intended usage/meaning, and the proper calculation of the dependency graph tied to the Callspec2 objects instead of the Metafunc object.
I think that test reordering might also need to be touched.

But with a feature that allows fixtures in parametrization and proper dependency calculation writing a plugin the behaves similar to pytest-cases and offers a unified parametrization UX is relative simple. As already stated i have an experimental (internal) implementation for an replacement uses a heavily modified version of pytest-lazy-fixtures under the hood that mostly works but does some sketchy stuff to inject dependencies.

Proposed Changes

Following is a loose and incomplete list of changes/tasks i would propose to tackle this.

• Investigate and implement calculation of the complete and exact⁴ dependency graph instead of the current fixture closure. Then names_closure/fixturenames is the iteration in topological sort order, if this order does not exist the graph has an cycle and is invalid. This would also address function-scoped fixture run before session-scoped fixture #5303, params on Fixture are not always detected during test generation #11350 and maybe Wrong session scoped fixtures parametrization #2844.
  I am aware that this is computationally expensive. But I think it is either this or recursive algorithms further down the line.
• Do not share the FuncFixtureInfo object between all calls to a test and attach it to the callspecs instead of MetaFunc. This is similar to what is included in Support usefixtures with parametrize #11298.
• Extend Metafunc.parametrize to recalculate the dependency graph. For now this would be only to prune the graph in case of direct parametrization.
• Implement LazyFixture/FixtureRef. This can be a simple dataclass with the name of the fixture and an optional field id for parameter id generation.
• Extend/reimplement Metafunc.parametrize to recalculate the dependency graph by adding branches and iterate along it to discover all parametrizations.
  Ideally this is done none-recursive.
• Check if the dependency graphs can be used elsewhere. Reordering? Fixture Setup?
• Check how ordering in deeply parametrized dependencies of higher scopes is impacted.

I would like to reiterate that this post is intended as a starting point for a discussion and not as an definite 'this needs to happen' roadmap.
I would appreciate feedback, comments and any help in making this happen.

Footnotes

1. https://github.com/TvoroG/pytest-lazy-fixture ↩ ↩²

2. https://github.com/smarie/python-pytest-cases ↩ ↩² ↩³

3. my words, not the authors ↩

4. i.e. resolved to a single FixtureDef instead all of them. ↩

[RonnyPfannschmidt]

Thank you for collecting those detail, for some of those I have a idea on easing implementations

Unfortunately I can't do the elaboration in a timely manner, butter I'll trying to give it an hour tomorrow

[jgersti]

I am working on the first point I proposed above but my time this winter is unfortunatly very limited. I will try to squeeze in some time to make a PR so I can get some feedback if this is a direction worth pursuing.

One implication is that some errors (cyclic dependencies and wrong scopes) now also need to be handled during collection time (and I have no idea how to properly report these)