jit: resolve a walk's coordinates from the frame and operand offset that produced them; gate the vendored CPython suite

[youknowone]

Five commits on top of origin/main. Four of them are one story: a
coordinate is only meaningful in the frame, and at the operand offset, that
produced it. The fifth is the gate that surfaced all of them.

The gate

check.py grows a cpython-suite stage that runs pyre/cpython_tests/run.py
against baseline.json (103 modules, ~300s, dynasm). The suite already
existed; nothing gated it, so a module going PASS -> FAIL was invisible
outside a manual run. Every defect below was found by turning it on.

Three wrong-frame reads

A full-body walk that descends into an inlined callee carries two sets of
metadata: the outer portal frame's, and the callee's. Interpreting a
callee-scoped quantity — a vable slot index, a JitCode offset, a Python pc —
with outer-frame metadata reads a real value from the wrong frame, which is
why these are miscompiles rather than aborts.

1. getarrayitem_vable_* / setarrayitem_vable_* slot split
   (vable_ops.rs). The locals/stack split used the outer sym's nlocals,
   so a callee's STORE_FAST was classified as a stack push and never
   recorded; the blackhole then read NULL for that local.
   test.test_datetime showed it as _find_ti's
   bisect_right(self.lt[dt.fold], ts) raising missing 1 required positional argument.

2. In-flight FOR_ITER identity (diag.rs). The stash paired the
   callee's op_pc with the outer JitCode's index, so
   inflight_foriter_body_pc resolved a callee offset through the caller's pc
   tables and answered with a Python pc belonging to neither loop. A callee
   loop's item was stashed under an identity no resume could match, and a
   caller loop resolving to the same pc had its own entry replaced.

3. Nested-inline last_instr (inline_call.rs). In f -> g -> h, the
   coordinate published onto the outer frame while a residual runs came from
   mapping the intermediate callee's op.pc through the portal's tables.
   Observable as sys._getframe(2).f_lineno reporting the driver's def-body
   line as well as the real call line — pinned by
   parity_tests/nested_inline_caller_lineno.py.

fbw_mode.inline_subwalk is not the discriminant for "am I in a callee
frame": it is also set for a canonical-helper descent and a recursive root
closure, neither of which resolves a callee. Frame layout keys off
ctx.callee_shadow; JitCode identity keys off
ctx.inline_callee_consts.jitcode_index — the same resolution
build_multi_frame_miframe already applies to its innermost frame.

The wrong operand offset

reconstructed_all_ref_call_stack read the residual op's Ref var-list at
operand offset 1. That holds for the Ref-only shape iRd>r, but the
method-form CALL helpers lower through iIRd>r, whose leading Int list is
variable-width — dispatch_residual_call_iIRd_kind reads its own Ref list at
1 + i_width. At offset 1 the read landed on the Int list's length byte
and resolved its register indices through the Ref bank.

The composed stack passed validation anyway, because the composition
self-corrects its height:

prefix_len = vstack_boxes.len() - fresh.len()   // absorbs any fresh.len()
stack      = vstack_boxes[..prefix_len] ++ fresh

stack.len() == vstack_boxes.len() for any fresh, so the flush's
depth_at_py_pc check could not fail. For p[0] inside a for p in ... body
the committed stack was [iterator, p, iterator], and the interpreter
re-executed the subscript with the loop's iterator as the index:

re.compile("|".join("%d" % x for x in range(2000)))
# TypeError: list indices must be integers or slices, not list_iterator
#   _compiler.py:504 _get_charset_prefix -> op, av = p[0]

which failed test.test_re. The offset now comes from the op's argcodes,
walking the widths of blackhole.py:112-157 — the same walk decode_op_at
performs. An op declaring no Ref list declines to the legacy replay.

Audited: the other 22 hard-coded offset-1 reads are reached only from
dispatch_residual_call_iRd_kind, where 1 is correct.
reconstructed_all_ref_call_stack was the single site reachable from both
dispatchers, because it hangs off try_walker_inline_resolved_user_call.

The iterator __setstate__ cursor

Parity coverage for the ssize_t boundary of the cursor, continuing #1074.

Testing

• pyre/extra_tests/parity_tests/run.py — all pass, including the two new
  fixtures (nested_inline_caller_lineno.py,
  foriter_body_call_abort_operand_stack.py) on cpython, dynasm and cranelift.
• cargo test -p pyre-jit-trace — 357 passed, 0 failed. Two new unit tests
  pin the FOR_ITER identity and the argcode-derived offset; both were checked
  green -> red -> green by negating their halves in place.
• check.py — cpython-suite dynasm PASS, 103 modules. test.test_re and
  test.test_datetime both pass.

Known base red, not from this branch: synth/pypy_type_surface regresses
bridges_compiled 5 -> 102 on all three backends at this merge-base. It is
the Cls.__name__ fold guarding a raw null w_class on getset_descriptor,
fixed by #1106, which is not in this base. Not re-recorded.

Branch note

This force-update drops 843de7569dc ("send pair[i] on the arity-2 tuple
specialisations back to the residual") from the branch. That commit removed the
subscript fold because it miscompiled test.test_datetime; the first commit
here fixes that miscompile at its root, and test.test_datetime passes with
the fold in place, so the workaround is no longer needed. The narrower
SpecialisedPairKind::Object decline that is already on main is untouched.

🤖 Generated with Claude Code

Summary by CodeRabbit

• New Features

  • Added optional CPython suite validation on Darwin ARM64, including baseline comparisons and timeout reporting.
  • Added --no-cpython-suite to skip CPython suite validation when needed.

• Bug Fixes

  • Improved JIT behavior for nested inlining, local-variable handling, iterator operations, caller line reporting, and attribute access.
  • Improved coordinate tracking across inlined calls to prevent incorrect runtime results.

• Tests

  • Added coverage for nested inlining, iterator state restoration, guard failures, and iterator dispatch behavior.

[coderabbitai]

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: Organization UI

Review profile: ASSERTIVE

Plan: Pro Plus

Run ID: b97e1517-4890-4b52-b6d4-d17a9ec8a30b

📥 Commits

Reviewing files that changed from the base of the PR and between f008c71 and d7eb409.

📒 Files selected for processing (2)

• pyre/pyre-jit-trace/src/jitcode_dispatch/mod.rs
• pyre/pyre-jit-trace/src/jitcode_dispatch/tests.rs

---

Walkthrough

The PR adds a Darwin ARM64 CPython suite gate and updates inline JIT frame, coordinate, FOR_ITER, and operand handling. It also adds parity tests for inline state, caller line numbers, and Python 3.14 iterator state boundaries.

Changes

CPython suite validation

Layer / File(s)	Summary
CPython suite gate pyre/check.py	The check runner adds Darwin ARM64 baseline configuration, bounded suite execution, output validation, failure reporting, --no-cpython-suite, and main-runner integration.

Inline JIT correctness

Layer / File(s)	Summary
Inline frame identity and locals pyre/pyre-jit-trace/src/jitcode_dispatch/mod.rs, bridge_subwalk.rs, inline_call.rs, vable_ops.rs	Inline shadows retain the active callee code object. Active-frame local boundaries, operand recovery, folded stores, and method-form LOAD_ATTR handling use frame-specific metadata.
Coordinate and operand reconstruction pyre/pyre-jit-trace/src/jitcode_dispatch/{inline_call.rs,diag.rs,residual_call.rs,specialize.rs,mod.rs}	Nested inline calls preserve inherited caller coordinates. FOR_ITER resolution uses the current walk context and owning JIT code identity.
FOR_ITER dispatch and validation pyre/pyre-jit-trace/src/jitcode_dispatch/tests.rs, pyre/extra_tests/parity_tests/{inline_callee_locals_across_guard.py,nested_inline_caller_lineno.py}	Tests validate inline FOR_ITER coordinates, callee locals across guard failures, repeated results, and nested caller line reporting.

Iterator state parity

Layer / File(s)	Summary
Iterator state boundary coverage pyre/extra_tests/parity_tests/iterator_setstate_python314.py	The test covers overflowing values, exact OverflowError messages, ssize_t limits, iterator clamping, and generic iterator boundary behavior.

Estimated code review effort: 4 (Complex) | ~60 minutes

Sequence Diagram(s)

sequenceDiagram
  participant Check
  participant Dynasm
  participant CPythonSuite
  Check->>Dynasm: run synthetic benchmarks
  Check->>CPythonSuite: run vendored CPython tests
  CPythonSuite-->>Check: return module results and output
  Check->>Check: enforce suite and module timeouts

Loading

Possibly related PRs

• youknowone/pyre#53: Adds the synthetic parity infrastructure extended by this CPython suite validation.
• youknowone/pyre#319: Addresses inline callee identity through code_ptr metadata.
• youknowone/pyre#999: Updates the CPython module baselines consumed by this validation.

Poem

A rabbit watched the JIT hop bright,
Through callee frames and lines of light.
CPython tests now guard the gate,
While iterators learn their state.
“OK!” squeaked Bun beneath the moon.

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title clearly summarizes the two main changes: JIT coordinate resolution and gating the vendored CPython suite.
Docstring Coverage	✅ Passed	Docstring coverage is 100.00% which is sufficient. The required threshold is 80.00%.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

✨ Finishing Touches

📝 Generate docstrings

• Create stacked PR
• Commit on current branch

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch rbigint

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 624d547875

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[chatgpt-codex-connector]

Preserve the frame identity through FOR_ITER delivery

When an inlined callee's FOR_ITER body has the same numeric Python offset as a loop in the outer live frame, this conversion discards jitcode_index and returns only that offset. The downstream stash matching in fbw_state.rs consequently aliases the two loops, and deliver_inflight_foriter_item validates and pushes the callee's item against the outer frame, allowing a guard abort to resume the caller with the wrong item/stack. Keep the JitCode/frame identity through matching and delivery, or refuse legacy delivery for a non-root frame.

AGENTS.md reference: AGENTS.md:L32-L42

Useful? React with 👍 / 👎.

[chatgpt-codex-connector]

Run the CPython gate only once on macOS CI

In the inspected .github/workflows/pyre-ci.yml, pyre-check-macos reuses the shared step that invokes default pyre/check.py, while the separate cpython-tests job already runs pyre/cpython_tests/run.py with the same dynasm baseline. Making this stage unconditional by default on darwin-arm64 therefore executes the roughly five-minute, 104-module gate twice on every CI run, doubling the expensive macOS work without adding coverage; disable it in one lane or remove the redundant standalone job.

Useful? React with 👍 / 👎.

[coderabbitai]

Caution

Some comments are outside the diff and can’t be posted inline due to platform limitations.

⚠️ Outside diff range comments (1)

pyre/pyre-jit-trace/src/jitcode_dispatch/inline_call.rs (1)

4083-4187: 🎯 Functional Correctness | 🟠 Major | 🏗️ Heavy lift

Gate nested fresh-coordinate derivation on the non-subwalk case.

When ctx.fbw_mode.inline_subwalk is true, ctx.fbw_mode.snapshot_sym remains the outer/root sym while call_site_py_pc uses that sym’s jitcode metadata with the callee’s op.pc. If collect_outer_active_boxes returns no boxes at any nested inline Call, this branch can map the nested offset through the root JitCode. Use the same !ctx.fbw_mode.inline_subwalk guard as the abort-coordinate path.

🤖 Prompt for AI Agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@pyre/pyre-jit-trace/src/jitcode_dispatch/inline_call.rs` around lines 4083 -
4187, Restrict the fresh CALL-site coordinate derivation in the
outer-active-boxes initialization to the non-subwalk case, using the same
!ctx.fbw_mode.inline_subwalk condition as the abort-coordinate path. When
inline_subwalk is true, always inherit ctx.outer_active_boxes and its associated
coordinate fields instead of calling collect_outer_active_boxes or mapping op.pc
through snapshot_sym metadata.

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Outside diff comments:
In `@pyre/pyre-jit-trace/src/jitcode_dispatch/inline_call.rs`:
- Around line 4083-4187: Restrict the fresh CALL-site coordinate derivation in
the outer-active-boxes initialization to the non-subwalk case, using the same
!ctx.fbw_mode.inline_subwalk condition as the abort-coordinate path. When
inline_subwalk is true, always inherit ctx.outer_active_boxes and its associated
coordinate fields instead of calling collect_outer_active_boxes or mapping op.pc
through snapshot_sym metadata.

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Organization UI

Review profile: ASSERTIVE

Plan: Pro Plus

Run ID: 5d9d00a7-91f9-4c3b-bd3d-cd8a05110ff6

📥 Commits

Reviewing files that changed from the base of the PR and between d98cd90 and 624d547.

📒 Files selected for processing (13)

• pyre/check.py
• pyre/extra_tests/parity_tests/foriter_body_call_abort_operand_stack.py
• pyre/extra_tests/parity_tests/inline_callee_locals_across_guard.py
• pyre/extra_tests/parity_tests/iterator_setstate_python314.py
• pyre/extra_tests/parity_tests/nested_inline_caller_lineno.py
• pyre/pyre-jit-trace/src/jitcode_dispatch/bridge_subwalk.rs
• pyre/pyre-jit-trace/src/jitcode_dispatch/diag.rs
• pyre/pyre-jit-trace/src/jitcode_dispatch/inline_call.rs
• pyre/pyre-jit-trace/src/jitcode_dispatch/mod.rs
• pyre/pyre-jit-trace/src/jitcode_dispatch/residual_call.rs
• pyre/pyre-jit-trace/src/jitcode_dispatch/specialize.rs
• pyre/pyre-jit-trace/src/jitcode_dispatch/tests.rs
• pyre/pyre-jit-trace/src/jitcode_dispatch/vable_ops.rs

[coderabbitai]

Note

GitHub couldn't provide a complete incremental comparison for this pull request, so CodeRabbit is performing a full review instead. This review may take a little longer.

[github-actions]

🤖 Codex parity review

Static analysis of this diff vs the local RPython/PyPy sources (commit d7eb409).
Updated: 2026-08-09T06:43:37.932Z

Files in the reviewed diff

pyre/check.py
pyre/extra_tests/parity_tests/inline_callee_locals_across_guard.py
pyre/extra_tests/parity_tests/iterator_setstate_python314.py
pyre/extra_tests/parity_tests/nested_inline_caller_lineno.py
pyre/pyre-jit-trace/src/jitcode_dispatch/bridge_subwalk.rs
pyre/pyre-jit-trace/src/jitcode_dispatch/diag.rs
pyre/pyre-jit-trace/src/jitcode_dispatch/inline_call.rs
pyre/pyre-jit-trace/src/jitcode_dispatch/mod.rs
pyre/pyre-jit-trace/src/jitcode_dispatch/residual_call.rs
pyre/pyre-jit-trace/src/jitcode_dispatch/specialize.rs
pyre/pyre-jit-trace/src/jitcode_dispatch/tests.rs
pyre/pyre-jit-trace/src/jitcode_dispatch/vable_ops.rs

1. Regressions to PyPy parity introduced by this patch

None.

2. Other mismatches introduced by this patch

None.

3. Pre-existing mismatches (already present before this patch)

• pyre/pyre-jit-trace/src/jitcode_dispatch/inline_call.rs:4035 ↔ rpython/jit/metainterp/pyjitpl.py:2445: pyre still permits a “single-frame collapse” that resumes by re-executing the call; RPython always creates and pushes an MIFrame for perform_call. This limitation predates the patch.

• pyre/pyre-jit-trace/src/jitcode_dispatch/mod.rs:395 ↔ pypy/interpreter/pyframe.py:84: pyre retains inline locals in CalleeLocalsShadow HashMap side tables, whereas PyPy owns locals/cells/stack directly in each frame’s locals_cells_stack_w array. The maps predate this patch.

4. Structural adaptations

• pyre/pyre-jit-trace/src/jitcode_dispatch/mod.rs:416 ↔ pypy/interpreter/pyframe.py:98: adding a Rust CodeObject* to the shadow is the Rust representation of PyPy’s per-frame pycode; it restores per-frame code/locals ownership rather than changing Python semantics.

• pyre/extra_tests/parity_tests/iterator_setstate_python314.py:164 ↔ pypy/objspace/std/iterobject.py:40: the test intentionally targets CPython 3.14 iterator __setstate__ rules (strict integer acceptance and producer-specific cursor clamping), while PyPy’s generic iterator uses space.int_w and stores most positive cursors directly. This is an explicit Python-version compatibility adaptation.

[coderabbitai]

Caution

Some comments are outside the diff and can’t be posted inline due to platform limitations.

⚠️ Outside diff range comments (1)

pyre/pyre-jit-trace/src/jitcode_dispatch/inline_call.rs (1)

4115-4186: 🎯 Functional Correctness | 🟠 Major | ⚡ Quick win

Preserve inherited caller state for nested inline subwalks.

When ctx.outer_active_boxes.is_empty() and ctx.fbw_mode.inline_subwalk is true, this branch maps the intermediate callee op.pc through ctx.fbw_mode.snapshot_sym::jitcode. That produces an invalid caller coordinate and can publish the wrong last_instr or resume header; skip collapse-box capture here and use the inherited ctx fields plus inherited_caller_py_pc instead.

Proposed fix

-    ) = if ctx.outer_active_boxes.is_empty() {
+    ) = if ctx.outer_active_boxes.is_empty() && !ctx.fbw_mode.inline_subwalk {

Add a nested-inline regression case where the outer caller has no active boxes. Before commit, run cargo check --features dynasm, cargo test --features dynasm, and all eight required benchmarks.

🤖 Prompt for AI Agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@pyre/pyre-jit-trace/src/jitcode_dispatch/inline_call.rs` around lines 4115 -
4186, When ctx.outer_active_boxes.is_empty() and ctx.fbw_mode.inline_subwalk is
true, bypass the snapshot_sym/jitcode call-site coordinate derivation and
collect_outer_active_boxes call; instead preserve the inherited ctx outer fields
together with inherited_caller_py_pc. Add a nested-inline regression case
covering an outer caller with no active boxes, then run the requested dynasm
checks, tests, and benchmarks.

Source: Coding guidelines

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Outside diff comments:
In `@pyre/pyre-jit-trace/src/jitcode_dispatch/inline_call.rs`:
- Around line 4115-4186: When ctx.outer_active_boxes.is_empty() and
ctx.fbw_mode.inline_subwalk is true, bypass the snapshot_sym/jitcode call-site
coordinate derivation and collect_outer_active_boxes call; instead preserve the
inherited ctx outer fields together with inherited_caller_py_pc. Add a
nested-inline regression case covering an outer caller with no active boxes,
then run the requested dynasm checks, tests, and benchmarks.

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Organization UI

Review profile: ASSERTIVE

Plan: Pro Plus

Run ID: 453949ed-84b4-4322-9b98-a43140b36e00

📥 Commits

Reviewing files that changed from the base of the PR and between e5eff81 and f008c71.

📒 Files selected for processing (12)

• pyre/check.py
• pyre/extra_tests/parity_tests/inline_callee_locals_across_guard.py
• pyre/extra_tests/parity_tests/iterator_setstate_python314.py
• pyre/extra_tests/parity_tests/nested_inline_caller_lineno.py
• pyre/pyre-jit-trace/src/jitcode_dispatch/bridge_subwalk.rs
• pyre/pyre-jit-trace/src/jitcode_dispatch/diag.rs
• pyre/pyre-jit-trace/src/jitcode_dispatch/inline_call.rs
• pyre/pyre-jit-trace/src/jitcode_dispatch/mod.rs
• pyre/pyre-jit-trace/src/jitcode_dispatch/residual_call.rs
• pyre/pyre-jit-trace/src/jitcode_dispatch/specialize.rs
• pyre/pyre-jit-trace/src/jitcode_dispatch/tests.rs
• pyre/pyre-jit-trace/src/jitcode_dispatch/vable_ops.rs