[self-tests] new test_too_many_open_files fails

[stanislavlevin]

Recently added (virtualenv 20.33.0) test_too_many_open_files fails:

========================================================= test session starts ==========================================================
platform linux -- Python 3.12.11, pytest-8.4.1, pluggy-1.6.0
cachedir: .tox/py3/.pytest_cache
Using --randomly-seed=2669281539
rootdir: /usr/src/RPM/BUILD/python3-module-virtualenv-20.33.0
configfile: pyproject.toml
plugins: time-machine-2.16.0, env-1.1.5, mock-3.14.1, randomly-3.16.0, timeout-2.4.0, flaky-3.8.1
timeout: 600.0s
timeout method: signal
timeout func_only: False
collected 1 item

tests/unit/test_file_limit.py F                                                                                                  [100%]

=============================================================== FAILURES ===============================================================
_______________________________________________________ test_too_many_open_files _______________________________________________________
tmp_path = PosixPath('/usr/src/tmp/pytest-of-builder/pytest-12/test_too_many_open_files0')

    @pytest.mark.skipif(sys.platform == "win32", reason="resource module not available on Windows")
    def test_too_many_open_files(tmp_path):
        """
        Test that we get a specific error message when we have too many open files.
        """
        import resource  # noqa: PLC0415
    
        soft_limit, hard_limit = resource.getrlimit(resource.RLIMIT_NOFILE)
        if soft_limit > 1024:
            pytest.skip("soft limit for open files is too high to reliably trigger the error")
    
        # Lower the soft limit to a small number to trigger the error
        try:
            resource.setrlimit(resource.RLIMIT_NOFILE, (32, hard_limit))
        except ValueError:
            pytest.skip("could not lower the soft limit for open files")
    
        # Keep some file descriptors open to make it easier to trigger the error
        fds = []
        try:
            fds.extend(os.open(os.devnull, os.O_RDONLY) for _ in range(20))
    
            with pytest.raises(SystemExit) as excinfo:
>               cli_run([str(tmp_path / "venv")])

excinfo    = <ExceptionInfo for raises contextmanager>
fd         = 30
fds        = [11, 12, 13, 14, 15, 16, ...]
hard_limit = 524288
resource   = <module 'resource' from '/usr/lib64/python3.12/lib-dynload/resource.cpython-312.so'>
soft_limit = 1024
tmp_path   = PosixPath('/usr/src/tmp/pytest-of-builder/pytest-12/test_too_many_open_files0')

tests/unit/test_file_limit.py:34:

...


cls = <class 'virtualenv.discovery.py_info.PythonInfo'>
app_data = AppDataDiskFolder(/usr/src/tmp/pytest-of-builder/pytest-12/session-app-data0)
exe = '/usr/src/RPM/BUILD/python3-module-virtualenv-20.33.0/.tox/py3/bin/python'
env = environ({'HOME': '/usr/src', 'TMPDIR': '/usr/src/tmp', 'TERM': 'xterm', 'PATH': '/usr/src/RPM/BUILD/python3-module-vir...p/pytest-of-builder/pytest-12/session-app-data0', 'VIRTUALENV_NO_PERIODIC_UPDATE': '1', 'VIRTUALENV_NO_DOWNLOAD': '1'})
raise_on_error = True, ignore_cache = False

    def from_exe(cls, app_data, exe, env=None, raise_on_error=True, ignore_cache=False):  # noqa: FBT002, PLR0913
        env = os.environ if env is None else env
        result = _get_from_cache(cls, app_data, exe, env, ignore_cache=ignore_cache)
        if isinstance(result, Exception):
            if raise_on_error:
>               raise result
E               RuntimeError: failed to query /usr/src/RPM/BUILD/python3-module-virtualenv-20.33.0/.tox/py3/bin/python with code 24 err: 'Too many open files'

app_data   = AppDataDiskFolder(/usr/src/tmp/pytest-of-builder/pytest-12/session-app-data0)
cls        = <class 'virtualenv.discovery.py_info.PythonInfo'>
env        = environ({'HOME': '/usr/src', 'TMPDIR': '/usr/src/tmp', 'TERM': 'xterm', 'PATH': '/usr/src/RPM/BUILD/python3-module-vir...p/pytest-of-builder/pytest-12/session-app-data0', 'VIRTUALENV_NO_PERIODIC_UPDATE': '1', 'VIRTUALENV_NO_DOWNLOAD': '1'})
exe        = '/usr/src/RPM/BUILD/python3-module-virtualenv-20.33.0/.tox/py3/bin/python'
ignore_cache = False
raise_on_error = True
result     = RuntimeError("failed to query /usr/src/RPM/BUILD/python3-module-virtualenv-20.33.0/.tox/py3/bin/python with code 24 err: 'Too many open files'")

.tox/py3/lib/python3/site-packages/virtualenv/discovery/cached_py_info.py:35: RuntimeError

[gaborbernat]

CC @esafak

[esafak]

What platform and operating system is this on? Got any public links to runs?

[gaborbernat]

RPM seems to indicate to me that it's some kind of Fedora variant, or Red Hat 🤔

[esafak]

I'm curious why RedHat would issue RuntimeError instead of SystemExit. I can add RuntimeError to the expected list but then we won't have an exit code to test :|

[frenzymadness]

I have a similar issue in Fedora Linux during the RPM build with Python 3.14 but the exception raised is different:

_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = <virtualenv.util.lock._CountedFileLock object at 0x7f46d4a691d0>

    def _acquire(self) -> None:
        ensure_directory_exists(self.lock_file)
        open_flags = os.O_RDWR | os.O_TRUNC
        if not Path(self.lock_file).exists():
            open_flags |= os.O_CREAT
>       fd = os.open(self.lock_file, open_flags, self._context.mode)
E       OSError: [Errno 24] Too many open files: '/tmp/pytest-of-mockbuild/pytest-0/session-app-data0/py_info/2/31f2aee4e71d21fbe5cf8b01ff0e069b9275f58929596ceb00d14d90e3e16cd6.lock'

open_flags = 514
self       = <virtualenv.util.lock._CountedFileLock object at 0x7f46d4a691d0>

/usr/lib/python3.14/site-packages/filelock/_unix.py:42: OSError

The beginning of the output is:

            with pytest.raises(SystemExit) as excinfo:
>               cli_run([str(tmp_path / "venv")])

excinfo    = <ExceptionInfo for raises contextmanager>
fd         = 31
fds        = [12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31]
hard_limit = 524288
resource   = <module 'resource' from '/usr/lib64/python3.14/lib-dynload/resource.cpython-314-x86_64-linux-gnu.so'>
soft_limit = 1024
tmp_path   = PosixPath('/tmp/pytest-of-mockbuild/pytest-0/test_too_many_open_files0')

[esafak]

OSError actually makes more sense than the other two; I'll add that one. Let's see if users report any others...

[frenzymadness]

Side note: The reporter appears to be using ALT Linux.

[stanislavlevin]

yes, ALTLinux.

[stanislavlevin]

This test hardcodes the exact number of file descriptors (32 is soft limit and 20 for padding) to trigger an exception on locking, but in my case this causes exception in different part of the code. Not sure about this magic number tho.

[esafak]

Which part of the code? Try #2936

[stanislavlevin]

Which part of the code? Try #2936

Please check the stacktrace I put above.
Maybe I'm wrong but catching RuntimeError in the test looks incorrect and leads to testing nothing because the exception is not triggered by the fix from the original PR (https://github.com/pypa/virtualenv/pull/2922/files#diff-3546cc8bf25dd21397e65a5b34ab6e697bc707808b520a218acd39556c5319caR32-R36) but some other random code.

[esafak]

I think you have a point. What if we expanded the list of exceptions caught in the source as in the test, and if they correspond to "too many open files", we can raise a singular OSError to catch in the test?