Re-enable Move Subinterpreter test for free-threaded Python 3.14

[rwgk]

Continuation of work done initially under PRs #5934 and #5933

The write-up below is a Cursor-generated.

In an earlier draft of the below, Cursor asked "Could this be a bug in cpython?" Cursor was offering to generate a minimal C-only reproducer. We can come back to that if we want to.

__

Investigation: Free-Threaded Python 3.14 Hang in "Move Subinterpreter" Test

To: @b-pass
From: Investigation with Cursor AI assistance
Date: December 20, 2025
Re: PR #5933 - Root cause of Py_EndInterpreter() hang on free-threaded Python 3.14.2

---

Executive Summary

We've isolated the exact cause of the "Move Subinterpreter" test hang on free-threaded Python 3.14.2. The issue is not in gil_safe_call_once_and_store, the internals capsule destructors, or any cleanup code.

The root cause is a single line in py::subinterpreter::create():

PyThreadState_Swap(prev_tstate);  // subinterpreter.h:124

When this is called after PyThreadState_DeleteCurrent() during subinterpreter creation, it leaves the system in a state where later calling Py_EndInterpreter() from a different thread causes a deadlock.

---

Background

The Failing Test

The "Move Subinterpreter" test (test_subinterpreter.cpp:94-119) does:

1. Creates a subinterpreter on the main thread
2. Activates it, imports modules, deactivates
3. Spawns a worker thread that:
   • Activates the same subinterpreter
   • Imports a module
   • Destroys the subinterpreter (sub.reset())
4. Joins the worker thread

This test passes on:

• Default (non-free-threaded) Python 3.14.2 ✅
• Free-threaded Python 3.14.0t ✅ (sporadically fails on macOS)

This test hangs on:

• Free-threaded Python 3.14.1t ❌
• Free-threaded Python 3.14.2t ❌

Prior Findings

A pure C reproducer (move_subinterpreter_redux.c) that mimics the same pattern using only CPython C API passes on both 3.14.0t and 3.14.1t. This indicated the issue was in pybind11's internals, not CPython itself.

---

Investigation Methodology

We systematically created minimal test cases, each removing one aspect of pybind11's subinterpreter handling, until we found what causes the hang.

Test Matrix

Test Case	Key Difference	Result
move_subinterpreter_redux.c	Pure C, no pybind11	✅ Pass
debug_pure_c_with_pb11_main.cpp	pybind11 main interpreter, pure C subinterpreter	✅ Pass
debug_no_swap_back.cpp	Like above, but don't swap back after creation	✅ Pass
debug_with_swap_back.cpp	Add PyThreadState_Swap(prev_tstate) after creation	❌ Hang
debug_no_get_internals.cpp	Skip get_internals() call	❌ Hang
debug_no_num_interp.cpp	Skip get_num_interpreters_seen()++	❌ Hang
Any test using py::subinterpreter::create()	Uses full pybind11 code	❌ Hang

The critical finding: The only difference between passing and failing tests is PyThreadState_Swap(prev_tstate) after PyThreadState_DeleteCurrent().

---

Root Cause Analysis

The Problematic Code Path

In subinterpreter.h, the create() function does:

static subinterpreter create(PyInterpreterConfig const &cfg) {
    error_scope err_scope;
    subinterpreter result;
    {
        // Activate main interpreter to hold its GIL
        subinterpreter_scoped_activate main_guard(main());

        auto *prev_tstate = PyThreadState_Get();  // Save main's tstate

        // Create subinterpreter - now creation_tstate_ is current
        Py_NewInterpreterFromConfig(&result.creation_tstate_, &cfg);

        result.istate_ = result.creation_tstate_->interp;
        detail::get_num_interpreters_seen() += 1;
        detail::get_internals();  // Initialize internals for subinterpreter

        // Clean up creation tstate (3.14+ style)
        PyThreadState_Clear(result.creation_tstate_);
        PyThreadState_DeleteCurrent();  // Thread state is now NULL

        // Switch back to main interpreter
        PyThreadState_Swap(prev_tstate);  // <-- THIS CAUSES THE HANG
    }
    return result;
}

Why This Causes a Hang

On free-threaded Python 3.14.2, the sequence:

1. PyThreadState_DeleteCurrent() - deletes the subinterpreter's creation tstate
2. PyThreadState_Swap(prev_tstate) - swaps to main interpreter's tstate

...appears to leave some internal state inconsistent. Specifically, when later:

3. A different thread creates a new tstate for the subinterpreter
4. That thread calls Py_EndInterpreter()

...the Py_EndInterpreter() call deadlocks.

The Pure C Pattern That Works

The working C reproducer does this instead:

static void create_subinterpreter(void) {
    PyThreadState *creation_tstate = NULL;
    Py_NewInterpreterFromConfig(&creation_tstate, &cfg);
    sub_interp = creation_tstate->interp;

    // Clean up creation tstate
    PyThreadState_Clear(creation_tstate);
    PyThreadState_DeleteCurrent();
    // NOTE: Does NOT call PyThreadState_Swap() here
    // Thread state is left as NULL
}

The key difference: no PyThreadState_Swap() after PyThreadState_DeleteCurrent().

---

Minimal Reproducer

Here's the minimal code that demonstrates the issue:

// HANGS on free-threaded Python 3.14.2
#include <pybind11/embed.h>
#include <thread>

namespace py = pybind11;

static PyInterpreterState *sub_interp = nullptr;

int main() {
    py::scoped_interpreter guard{};
    
    // Create subinterpreter
    {
        PyThreadState *prev_tstate = PyThreadState_Get();  // Save main tstate
        
        PyInterpreterConfig cfg = {0};
        cfg.allow_threads = 1;
        cfg.check_multi_interp_extensions = 1;
        cfg.gil = PyInterpreterConfig_OWN_GIL;

        PyThreadState *creation_tstate = nullptr;
        Py_NewInterpreterFromConfig(&creation_tstate, &cfg);
        sub_interp = creation_tstate->interp;
        
        PyThreadState_Clear(creation_tstate);
        PyThreadState_DeleteCurrent();
        
        PyThreadState_Swap(prev_tstate);  // <-- REMOVE THIS LINE TO FIX
    }

    // Use subinterpreter on main thread
    {
        PyThreadState* tstate = PyThreadState_New(sub_interp);
        PyThreadState* old = PyThreadState_Swap(tstate);
        PyRun_SimpleString("import datetime");
        PyThreadState_Clear(tstate);
        PyThreadState_DeleteCurrent();
        PyThreadState_Swap(old);
    }

    // Destroy from worker thread - THIS HANGS
    std::thread([&]() {
        PyThreadState* tstate = PyThreadState_New(sub_interp);
        PyThreadState_Swap(tstate);
        PyThreadState_Clear(tstate);
        PyThreadState_DeleteCurrent();
        
        PyThreadState *destroy_tstate = PyThreadState_New(sub_interp);
        PyThreadState_Swap(destroy_tstate);
        Py_EndInterpreter(destroy_tstate);  // <-- HANGS HERE
    }).join();
    
    return 0;
}

To fix: Remove the PyThreadState_Swap(prev_tstate) line after creation.

---

Attempted Fixes

We tried several quick fixes, none of which fully worked:

Attempt 1: Skip PyThreadState_Swap(prev_tstate) on free-threaded Python

#if defined(Py_GIL_DISABLED) && PY_VERSION_HEX >= 0x030D0000
    (void) prev_tstate;  // Skip the swap
#else
    PyThreadState_Swap(prev_tstate);
#endif

Result: No longer hangs, but crashes with:

Fatal Python error: PyGILState_Release: auto-releasing thread-state, 
but no thread-state for this thread

The subinterpreter_scoped_activate main_guard(main()) destructor tries to call PyGILState_Release() but there's no current thread state after we skipped the swap.

Attempt 2: Create a fresh thread state for main instead of reusing prev_tstate

#if defined(Py_GIL_DISABLED)
    (void) prev_tstate;
    PyThreadState *fresh_main_tstate = PyThreadState_New(PyInterpreterState_Main());
    PyThreadState_Swap(fresh_main_tstate);
#else
    PyThreadState_Swap(prev_tstate);
#endif

Result: Segfault. The fresh tstate doesn't properly integrate with the saved state in main_guard.

Conclusion

The fix is not trivial because the subinterpreter_scoped_activate main_guard(main()) at line 85 saves state (gil_state_ and/or old_tstate_) that becomes stale after PyThreadState_DeleteCurrent(). A proper fix likely requires restructuring how create() manages the main interpreter's GIL, possibly:

1. Not using subinterpreter_scoped_activate inside create() on free-threaded Python
2. Using a different pattern that doesn't rely on swapping back to a saved thread state
3. Some other approach?

---

Files Referenced

• include/pybind11/subinterpreter.h - lines 79-127 (create() function)
• tests/test_with_catch/test_subinterpreter.cpp - lines 94-119 ("Move Subinterpreter" test)
• Debug test files:
  • debug_no_swap_back.cpp - passes
  • debug_with_swap_back.cpp - hangs

---

Appendix: Debug Output

Passing Test (no swap back)

[DEBUG] Starting scoped_interpreter...
[DEBUG] scoped_interpreter created
[DEBUG] Creating subinterpreter...
[DEBUG] Subinterpreter created, id=1
[DEBUG] Main thread: activating...
[DEBUG] Main thread: imported
[DEBUG] Spawning worker thread...
[DEBUG] Worker thread: started
[DEBUG] Worker thread: activated
[DEBUG] Worker thread: about to Py_EndInterpreter...
[DEBUG] Worker thread: Py_EndInterpreter done!
[DEBUG] Thread joined, test complete!
Exit code: 0

Failing Test (with swap back)

[DEBUG] Starting scoped_interpreter...
[DEBUG] scoped_interpreter created
[DEBUG] Creating subinterpreter...
[DEBUG] Called PyThreadState_Swap(prev_tstate) after DeleteCurrent
[DEBUG] Subinterpreter created, id=1
[DEBUG] Main thread: activating...
[DEBUG] Main thread: imported
[DEBUG] Spawning worker thread...
[DEBUG] Worker thread: started
[DEBUG] Worker thread: activated
[DEBUG] Worker thread: about to Py_EndInterpreter...
Exit code: 124  (timeout - hung in Py_EndInterpreter)

---

This investigation was conducted using local builds of Python 3.14.2 (default and free-threaded) from commit df793163d58.

---

📚 Documentation preview 📚: https://pybind11--5940.org.readthedocs.build/

[rwgk]

@b-pass, it'd be great if we could connect directly. Could you please email me under the email address you see with git log | grep rwgkio?

[rwgk]

@b-pass This is now a real PR. Could you take it from here?

[b-pass]

Simpler than expected, Python is doing a "stop-the-world" during Py_EndInterpreter. So before we can join the thread, we have to detach from any/every/all thread states (including those from different interpreters or, in this case, the main interpreter). So that's why removing the Swap worked, it left the main thread with no active interpreter.

The fix is easy, a GIL release call detaches the state. It's rather unexpected though, I think it makes sub-interpreters a little more awkward to use. I added a note in the docs about it.

I had a lot of trouble getting a usable nogil environment ... seems like pyconfig.h doesn't include properly from the deadsnakes PPA install unless I include <pyconfig.h> before <Python.h>, which seems wrong.

[b-pass]

This was changed in python/cpython#128639

[rwgk]

Thanks a lot @b-pass for figuring this out. I had Cursor explain to me why this works. I'll attach some of the stuff it explained to me.

I cannot approve here because technically it's my own PR. I'll go ahead and merge this.

[rwgk]

Cursor-generated:

---

When Was Stop-the-World Added to Py_EndInterpreter?

The Actual Breaking Commit

The commit that introduced STW in Py_EndInterpreter() is:

21914979335 gh-136003: Execute pre-finalization callbacks in a loop (GH-136004)
Author: Peter Bierma <[email protected]>
Date:   Thu Sep 18 08:29:12 2025 -0400

Not commit 08bea299bfd (which is about out-of-memory handling).

What Changed

The commit added make_pre_finalization_calls() which is called from Py_EndInterpreter():

// From Python/pylifecycle.c

void
Py_EndInterpreter(PyThreadState *tstate)
{
    // ...
    interp->finalizing = 1;

    // This call stops the world and takes the pending calls lock.
    make_pre_finalization_calls(tstate, /*subinterpreters=*/0);

    ASSERT_WORLD_STOPPED(interp);
    // ...
}

And inside make_pre_finalization_calls():

static void
make_pre_finalization_calls(PyThreadState *tstate, int subinterpreters)
{
    for (;;) {
        // ... do cleanup work ...

        /* Stop the world to prevent other threads from creating threads or
         * atexit callbacks. */
        PyMutex_Lock(&interp->ceval.pending.mutex);
        _PyEval_StopTheWorldAll(interp->runtime);  // <-- HERE!
        
        // Check if more work is needed...
        if (!should_continue) {
            break;  // Exit with world stopped
        }
        _PyEval_StartTheWorldAll(interp->runtime);
        PyMutex_Unlock(&interp->ceval.pending.mutex);
    }
    // Returns with world STOPPED
    ASSERT_WORLD_STOPPED(interp);
}

Key Points

1. Free-threading only: The _PyEval_StopTheWorldAll() call is inside #ifdef Py_GIL_DISABLED

2. Added in Python 3.14 development cycle: The commit is dated September 2025

3. Why it was added: To safely handle atexit callbacks, pending calls, and thread cleanup during interpreter finalization. Multiple threads could be creating new threads or callbacks, so STW ensures a clean shutdown.

4. The loop: Finalization now runs in a loop because atexit callbacks can spawn threads, threads can add pending calls, etc. STW is used to check "are we done yet?" safely.

Why Commit 08bea299bfd Was Misleading

That commit (gh-112729: Correctly fail when the process is out of memory) was a cherry-pick to the 3.14 branch that happened around the same time. It touched Python/pylifecycle.c but didn't introduce the STW behavior. The STW change was already in the codebase from the earlier gh-136003 commit.

Timeline

• 3.13: Free-threading introduced, but Py_EndInterpreter() didn't do STW
• 3.14 (Sep 2025): gh-136003 added STW to interpreter finalization
• 3.14.1+: The fix is needed for subinterpreter destruction on non-main threads

[rwgk]

Cursor-generated:

---

Why Bisect Found 08bea299bfd Instead of 21914979335

Different Branches

21914979335 → on `main` branch only
08bea299bfd → on `3.14` branch only

The bisect was likely done on the 3.14 branch (the release branch you were testing against), not main.

The 3.14 Branch Has Different Code

Looking at the 3.14 branch's Py_EndInterpreter():

// 3.14 branch version (simpler)
Py_EndInterpreter(PyThreadState *tstate)
{
    // ...
    wait_for_thread_shutdown(tstate);
    _Py_FinishPendingCalls(tstate);
    _PyAtExit_Call(tstate->interp);
    
    _PyEval_StopTheWorldAll(runtime);  // STW is here directly
    // ...
}

vs. main branch:

// main branch version (with the loop)
Py_EndInterpreter(PyThreadState *tstate)
{
    // ...
    make_pre_finalization_calls(tstate, 0);  // STW is inside this function
    // ...
}

The Actual Culprit on 3.14

On the 3.14 branch, STW in Py_EndInterpreter was added by:

cec4ddf23e5 [3.14] gh-128639: Don't assume one thread in subinterpreter finalization 
             with fixed daemon thread support (GH-134606) (GH-139050)
Date: Tue Oct 7 19:30:38 2025

And 08bea299bfd was cherry-picked at the same time (Oct 7 2025 19:30:23).

Why Bisect Picked 08bea299bfd

Most likely:

1. Both commits landed together in the same batch of cherry-picks
2. The bisect found 08bea299bfd because it was the first in that batch (by a few seconds or commit order)
3. The actual STW change was in cec4ddf23e5 but bisect identifies the first "bad" commit, which could be either if they landed together

Or:

1. Your test case specifically triggered something that 08bea299bfd changed (the _PyInterpreterState_New refactor)
2. The interaction between the new _PyInterpreterState_New calling convention AND the STW code caused the specific deadlock

Bottom Line

The bisect result is technically accurate—08bea299bfd is the first commit where your test failed on the 3.14 branch. But the root cause (STW in Py_EndInterpreter) was introduced by cec4ddf23e5 which landed in the same batch.

The fix (releasing the GIL before joining) addresses the STW issue regardless of which specific commit introduced it.