test_distutils_adoption.py in internet-less environment

[hroncok]

Hello,
in Fedora, we run the tests of setuptools in an environment without internet connection.

We already skip setuptools/tests/test_integration.py entirely, but recently setuptools/tests/test_distutils_adoption.py started to fail as well (when upgrading from 53.x to 54.x).

I suspect that #2582 re-added pyproject.toml, forcing pip to use build isolation which requires internet access. The tests now try to fetch wheel (as a build requirement) from PyPI when installing setuptools to a jaraco.envs environment.

Monkeypatching the VirtualEnv(jaraco.envs.VirtualEnv) class to use pip with the --no-build-isolation flag in the install method workarounds the problem.

We see similar attempts to fetch wheel in TestSphinxUploadDocs.test_sphinx_doc and test_pip_upgrade_from_source[None], but I have not yet figured out how to workaround those, except to rm pyproject.toml.

Is it desirable to make those tests work without access to real PyPI or should we skip them one by one in downstream instead?

[webknjaz]

I wonder if using a fixture like https://github.com/jazzband/pip-tools/blob/d6e1f9f/tests/conftest.py#L215-L225 would be helpful, or maybe a fixture setting PIP_FIND_LINKS env var (https://pip.pypa.io/en/latest/user_guide/#environment-variables).

We could add a gitignored directory like .wheel-cache/ that could be pre-populated with dists necessary for testing in envs like Fedora's CI, before running the tests. And normal setuptools tests wouldn't do that having access to the Internet. We'd have PIP_FIND_LINKS=/path-to/.wheel-cache and those dists would be used as a fallback, if present, whenever PyPI isn't available.

Alternatively, instead of having an in-tree gitignored location, we could just add a pytest option allowing the caller to point to any wheelhouse dir on the machine that they manage.

[webknjaz]

Interesting... Let's ask @pradyunsg — is there any way to make pip bypass external settings into isolated envs?

[webknjaz]

OTOH -i sets an index while -f sets --find-links which shouldn't be mutually exclusive. Could you try also using PIP_NO_INDEX=true?

[hroncok]

That makes the otherwise failed tests work, but breaks another test.

test_dist_fetch_build_egg
...
ERROR: Could not find a version that satisfies the requirement barbazquux-runner
ERROR: No matching distribution found for barbazquux-runner

However, I think that test should monkeypatch PIP_NO_INDEX and PIP_FIND_LINKS out of the environment.

[webknjaz]

Well, at least we confirmed that it has the potential to fix the issue. Of course, it's best to make a granular fixture for this. Let's wait to have some additional feedback from @jaraco though.

[webknjaz]

I think, the solution I'd like is to have some flag so you could just invoke the tests with something like pytest --offline-friendly and the internal fixtures would conditionally add --no-index stuff where it makes sense.

[jaraco]

Is it desirable to make those tests work without access to real PyPI or should we skip them one by one in downstream instead?

It is undesirable that these tests access PyPI. At the very least, there should be an option to skip such tests or for the tests to be automatically skipped when PyPI isn't available. Better would be for the tests to run fully in an isolated environment.

I'd like to find a solution that's maximally general and minimally invasive to each test.