Getting class-scoped parameters to work better

[ydirson]

context and original state

I have a set of inter-dependent parametrized tests originally set up as follows. They respectively test:

• installation of a system (where actual checks have to be done alongside a long-running task, so inside a single test function that performs installation)
• test that a number of things properly happen during first-boot (originally follows the same approach as for install, more on this later)
• upgrade of system to a new version (which is essentially similar to install, but has to be performed on a system that did have a successful first boot)

Edit: original post had a mistake explained in first followup, confusing content was removed.

The tests can be run on several version, which nicely map to test parameters, and I am able to name each and every dependencies for pytest-dependency to ensure tests are run in correct order:

import pytest

@pytest.mark.parametrize("iso_version", ("1", "2"))
@pytest.mark.dependency()
def test_install(iso_version):
    pass

@pytest.mark.parametrize("what", (
    pytest.param("1", marks=[pytest.mark.dependency(depends=["test_install[1]"])]),
    pytest.param("2", marks=[pytest.mark.dependency(depends=["test_install[2]"])]),
))
def test_firstboot(what):
    # do boot
    assert 1
    assert 2
    # do shutdown

@pytest.mark.parametrize("what,iso_version", [
    pytest.param("1", "2", marks=[pytest.mark.dependency(depends=["test_firstboot[1]"])]),
])
def test_upgrade(what, iso_version):
    pass

trying to follow best practice

The Fixtures HOWTO makes a good point that my test_firstboot would be better with an autouse fixture and 2 separate tests.

Refactoring gives the following for "first boot" . Note the switch of the what parameter to "class" scope, which looks like a natural way to keep the related tests together.

@pytest.mark.parametrize("what", (
    pytest.param("1", marks=[pytest.mark.dependency(depends=["test_install[1]"])]),
    pytest.param("2", marks=[pytest.mark.dependency(depends=["test_install[2]"])]),
),
                         scope="class")
class TestFirstboot:
    @pytest.fixture(autouse=True, scope="class")
    def booted_machine(self, what):
        # do boot
        yield
        # do shutdown
        
    def test_one(self, request, what):
        logging.info("parent: %s", id(request.node.parent))
        assert 1
    def test_two(self, request, what):
        logging.info("parent: %s", id(request.node.parent))
        assert 2

We can see all those tests have a single "Class" parent:

test_scopes1.py::test_install[1] PASSED                                                                                                                                                                     [ 14%]
test_scopes1.py::test_install[2] PASSED                                                                                                                                                                     [ 28%]
test_scopes1.py::TestFirstboot::test_one[1] 
--- live log call ---
INFO     root:test_scopes1.py:22 parent: 132558541321760
PASSED [ 42%]
test_scopes1.py::TestFirstboot::test_two[1] 
--- live log call ---
INFO     root:test_scopes1.py:25 parent: 132558541321760
PASSED [ 57%]
test_scopes1.py::TestFirstboot::test_one[2] 
--- live log call ---
INFO     root:test_scopes1.py:22 parent: 132558541321760
PASSED [ 71%]
test_scopes1.py::TestFirstboot::test_two[2] 
-- live log call ---
INFO     root:test_scopes1.py:25 parent: 132558541321760
PASSED [ 85%]
test_scopes1.py::test_upgrade[1-2] 

Wouldn't it make sense to have distinct TestFirstboot nodes, parametrized into TestFirstboot[1] and TestFirstboot[2]?

Having separate class nodes for different parameter sets, like TestFirstboot[1] and TestFirstboot[2], also seem to be necessary for any caching done at the class-fixture level: even when using the workaround, where they are different collected class nodes, we would need to get distinct nodeids for them.

depending on such multiple-assert test

Now the question is, how to adjust the dependency of test_upgrade to get the same effect as the original version.

Regardless of parameters, pytest-dependency currently only seems to support depending on tests, and although it can make sense to allow a dependency on a class in such a case with only effects-free tests, where it could be made an equivalent to "all tests collected below this node", it would not be reasonable to unable it systematically. Maybe such a behavior could be activated in pytest-dependency through an effect_free_tests or allow_dependency class marker?

A workaround is to specify as dependency one of the tests in the class

wrapping up

Before opening any issues - did I miss any existing feature to get this working without any tool evolution? Does all of this make sense to others?

[ydirson]

It turns out I messed up a bit when writing the original post's test code, and something got weird. The booted_machine lacked the self argument, and the what test parameter is not passed to the fixture.

(edited original post to fix that)