Write a PEP for a standard API for running test runners

[brettcannon]

(Hopefully) every project has tests. But the Python community doesn't have a standard way to know how to run an arbitrary project's test suite. This not only applies to new developers to a project but also people who want to run a project's test suite after downloading the sdist and building the wheel to make sure everything works fine.

Now this might not be worth it, but is there any interest in developing a PEP like https://www.python.org/dev/peps/pep-0517/ but for discovering and running tests? Or something simpler like how to list testing requirements like https://www.python.org/dev/peps/pep-0518/ and the command-line to run the tests (I guess I'm thinking along the lines of scripts from package.json, but making sure testing is specifically covered)? I know tox, nox, etc. often get used to execute stuff, but the fact I have to iterate more than one tool suggests to me that rallying around a way to specify how to run your test suite might be useful for people.

Anyway, at minimum, thanks for letting me dump this idea out of my brain. 😁

[RonnyPfannschmidt]

The standard for that will be tricky
Dependencies for tests can have multiple distinct dependencies for matrixes,

Additionally the test running entrypoint is unclear
As there is a number of options and ides may need to handle those distinctive

A standard for reporting test results and collection results is also going to be tricky as different test runners use vastly different mechanisms

A minimal standard could start at listing non optional test requirements and the cli entrypoint of the test runner/framework

[The-Compiler]

Reminds me of:

Joking aside: All projects I've encountered so far either run pytest directly (if very small); or use tox. I've not seen a single project in the wild using nox. I'm sure they exist, but at least in my experience, it's far from common. PyPI downloads for the last 30 days (tox, nox) might prove me wrong, but still, tox is more popular by factor 6 there.

That being said, I wonder if this could be a part of pyproject.toml indeed - but I'd really like to be able to specify "run tox to run my tests, it will take care of the rest" in there rather than having to specify my test dependencies again.

However, I'd be wondering what would be the consumer of that format then. As @RonnyPfannschmidt points out, tight IDE integration will be tightly coupled to a specific test runner (so an IDE would probably only need to know "this project uses unittest" vs. "this project uses pytest", the dependencies would probably be handled by a different mechanism, like other dependencies). For manual invocation, we lack a "meta-meta-testrunner" then. It seems weird for that tool to be pip, and a new tool introduces the problem from the XKCD above.

So, I'm not entirely sure what problem that would solve. I'd rather keep tox as the widely accepted quasi-standard it already is IMHO.

[RonnyPfannschmidt]

For the layer @brettcannon outlines,it would not replace tox/nox it would be a alternative to the command section

As is, unless there is a larger buy in/commitment from a large portion of the up/downstream ecosystem of test running in python this one will be considerably worse than the xkcd

[brettcannon]

Yep, no buy-in means it isn't going to happen (and so far no one is, so I'm not expecting this to go anywhere beyond me writing some Nox-based package to minimize some duplication I find with all automation tools).

And as for Nox usage, a decent chunk of the packaging ecosystem uses it as do I (e.g. pip, pipx, packaging, etc.). So while it might not be as popular as Tox, but it's still relevant.