enable pytest_plugins=('pytester',) for some test modules but not all?

[ev-br]

This is a usage question; if this is not the right venue for usage questions, please accept my apologies for the noise and please point me to the right medium, I'll repost there.

And I do not have a clean self-contained reproducer; am asking about a problem which only occurs under very specific circumstance with several moving parts.
I suspect it's me not doing something right with pytest plugins or plugin testing machinery, hence the question.

Full gory details: scipy/scipy_doctest#184
That thread is long, so I'll try to summarize below.

For context: I've a custom pytest plugin which wires itself into the pytest doctesting machinery, https://github.com/scipy/scipy_doctest

Self-tests for the plugin started failing with scipy 1.15.2. While it's possible that scipy itself does something bad, I suspect the problem is in how the plugin repository organizes its tests.

The observable side of the problem is an import error, where a doctesting a docstring from scipy.stats import levene fails inside the stdlib doctesting machinery, on, AFAICS, running exec('from scipy.stats import levene'). What fails to import is an implementation detail of one of scipy packages---I do not believe specifics matter because:

• the package imports fine outside of pytest
• the package only fails to import in very specific circumstances (below).

There are two test modules: module (1) is https://github.com/scipy/scipy_doctest/blob/main/scipy_doctest/tests/test_testmod.py and module (2) https://github.com/scipy/scipy_doctest/blob/main/scipy_doctest/tests/test_pytest_configuration.py

Both test modules import an "implementation module", which contains the doctest which imports scipy.

In the test run, an import error occurs in the module (2) but not module (1).
Module (1) tests are executed before those in module (2).

Module (2) does not use my pytest plugin, while module (1) does. Moreover, module (1) uses a global

pytest_plugins = ['pytester']

Experimentally, there are two ways to avoid the import error:

1. Add a scipy submodule import at global scope of module (2) [here's a PR with the exact import: https://github.com/MAINT : stop upper capping scipy scipy/scipy_doctest#187]

2. Alternatively, without importing scipy at global scope, manually comment out a test from module (1), test_module_cases (https://github.com/scipy/scipy_doctest/blob/main/scipy_doctest/tests/test_pytest_configuration.py#L26)

So, questions:

1. Am I doing something illegal with using the pytester fixture in some modules but not others?

2. Is it possible that pytest_plugins= ("pytester",) used in one test, somehow messes up some global state for subsequent tests? A possible tell is that the problem occurs only if two tests, one with pytester and the second without it, both import the same "implementation" module. If so, is there a way to selectively activate this fixture only for tests that need it? Or "unload" it after tests that need it have finished running?

3. Is there a sensible way to create tests for a doctest plugin on the fly without using the "pytester" fixture?

4. Or maybe I'm chasing shadows and there's some other obvious problem I should be looking for?

---

I fully realize it's a bad question if even explaing it takes a wall of text. However this is the best I got at the moment, and I'd very much appreciate any pointers.

Thanks,

Evgeni

[lucascolley]

Self-tests for the plugin started failing with scipy 1.5.2.

To clarify, it is SciPy 1.15.2, not 1.5.2

[ev-br]

Thanks for spotting the typo Lucas, have edited in the fix.

[RonnyPfannschmidt]

Pytester takes some localized snapshots of sys.modules to undo in process test running imports

https://github.com/pytest-dev/pytest/blob/main/src%2F_pytest%2Fpytester.py#L735 may need a extension to add user defined prefixes perhaps via entrypoint or a api

[ev-br]

Oh, thank you! I can confirm that adding "scipy" to the list of exclusions fixes the error we're seeing in the plugin.

Now, would you be willing to consider a pull request to either just add "scipy" to the exclusions, or add an API? And if so, what kind of API: would just exposing a pytester attribute, pytester.modules_to_snapshot which defaults to the existing list, ["zope", "readline"], would that be acceptable?

[RonnyPfannschmidt]

Id like a entrypoint as that would allow modules to register themselves

A quick hack adding scipy + a regression test for it would allow to defer the api design process

[ev-br]

Not sure how an entrypoint is going to look like, or what an entrypoint actually is. Mind expanding a bit, or giving me an example?

Re quick hack + regression test: a test is not easy, unfortunately. The problem does not show with just pytest-ing scipy, or doctesting its docstrings; the problem only appears with scipy + doctests + doctesting plugin + its test suite.