Recursion budget: a Python frame costs ~1.8 KB of C stack in release and ~9 KB in debug (PyPy ~0.56 KB); debug builds cannot import tempfile

[youknowone]

Symptom

A debug build cannot import tempfile:

$ target/debug/pyre-dynasm -c 'import tempfile'
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "lib-python/3/tempfile.py", line 43, in <module>
    import shutil as _shutil
  File "lib-python/3/shutil.py", line 10, in <module>
    import fnmatch
  File "lib-python/3/fnmatch.py", line 17, in <module>
    import re
  File "lib-python/3/re/__init__.py", line 126, in <module>
    from . import _compiler, _parser
  File "lib-python/3/re/_compiler.py", line 14, in <module>
    from . import _parser
  File "lib-python/3/re/_parser.py", line 15, in <module>
    from ._constants import *
RecursionError: maximum recursion depth exceeded

import re on its own is fine; sys.setrecursionlimit(5000) first makes the whole chain pass. Release builds are unaffected. Nothing is wrong with the import machinery — the seven-module chain simply does not fit in the budget a debug build has.

Measurement

sys.setrecursionlimit(N) then recursing until RecursionError:

build	N=1000	N=2000	N=4000	bytes of stack per Python frame
pyre debug (dynasm)	87	177	356	~9.0 KB
pyre release (dynasm and cranelift)	440	883	1588	~1.8 KB
PyPy 7.3.22	1396	2807	—	~0.56 KB
CPython 3.14.5	997	1997	3997	n/a (counts frames)

Depth scales linearly with the limit, so sys.setrecursionlimit is honoured and the formula is doing what it should. What differs is the constant: a pyre release frame costs ~3.2x the C stack of a translated-PyPy frame, and a debug frame ~16x. sys.getrecursionlimit() reports 1000 either way, so the number a user reads is 2.3x (release) or 11x (debug) more optimistic than the depth they actually get.

Mechanism

pyre/pyre-interpreter/src/stack_check.rs is a faithful port of rpython/rlib/rstack.py:42 stack_check + rpython/translator/c/src/stack.c:25: the budget is MAX_STACK_SIZE * (limit * 0.001) bytes with MAX_STACK_SIZE = 3 << 18 (768 KB), and sys.setrecursionlimit(N) calls pyre_stack_set_length_fraction(N * 0.001). So the byte budget matches an equivalent PyPy build exactly; only the per-frame consumption differs, and the module doc already says so:

Note that Rust interpreter frames are larger than RPython's translated-C frames, so the default recursion budget will exhaust sooner (in terms of Python-level call depth) than CPython or translated PyPy at the same sys.setrecursionlimit().

The gap is that this acknowledged deviation has grown large enough to break ordinary stdlib code in a debug build.

Why it is worth fixing rather than documenting

• A debug build is what local development, cargo test, and any interactive debugging session use. import tempfile is not an edge case — it comes in via shutil, and any script reaching for a temporary file hits it.
• 440 frames in release is itself below CPython's 997, so recursive user code that CPython accepts still fails on a release pyre at the default limit.
• The interpreter frame is the lever: at 768 KB the budget is already PyPy-identical, so raising MAX_STACK_SIZE would be a deliberate divergence from stack.h:8-19. Shrinking what an interpreter frame puts on the C stack (PyFrame::run / the eval loop / the call path) is the orthodox direction and buys both builds at once.

Possibly a recent regression, unconfirmed

Earlier on 2026-07-29 the same debug binary — same working tree, older base — imported tempfile successfully; after rebasing onto a newer main the identical script failed. The only variable that changed was the base, which picked up 0664030cf3 (#791, touches the import entry points), a510d58a73 (#856), 38994b93b7 (#834) and b8a00beb94 (#858). The old binary is gone, so this is a timing observation, not a bisect — #791 is the first thing to check if someone wants to attribute the last few frames.

Gate coverage

pyre/check.py stays green (dynasm 332/332, cranelift 332/332, wasm 329/329) because it exercises release binaries, where the chain still fits. No fixture imports tempfile, so nothing in the suite tracks how close the budget is to the edge — a fixture that imports it (or asserts a floor on the measured depth) would give the signal.

— reported by Claude