bench: record the measured jit-stats for three synthetic baselines - #1011
Merged
Conversation
check.py fails on origin/main across all three OS runners. Three recorded
baselines do not match what any platform measures.
exception_try_call_inlined_callee_raise.cranelift
loops_compiled 3 -> 2, guard_failures 402 -> 201
exception_try_call_inlined_callee_raise.wasm
loops_compiled 3 -> 2, loops_aborted 1 -> 0, guard_failures 821 -> 201
exc_in_loop_divzero_continue.wasm
bridges_compiled 0 -> 3, guard_failures 360 -> 603
The first two record the values every runner reads. The cranelift file
returns to the content #1003 replaced, and the wasm file, whose fields
#1009 first recorded, converges on that same content. Measured over
repeated runs of pyre-cranelift and pyre-wasm-runner, identical each time,
and CI reads 2 on ubuntu, macos and windows alike.
The third records a bridge count that rose from zero. #960 compiles
bridges for this fixture where none were compiled before; the new bridges
carry their own guards, so guard_failures rises with them. loops_compiled
stays at 3, so no loop was lost.
check.py --synthetic-only: cranelift 356/356, wasm 355/355.
The CPython suite gate is red for an unrelated reason and is untouched
here: test.test_asyncio went PASS -> TIMEOUT starting at #1003 and is a
behaviour change, not a recorded value.
Assisted-by: Claude
|
No actionable comments were generated in the recent review. 🎉 ℹ️ Recent review info⚙️ Run configurationConfiguration used: Organization UI Review profile: ASSERTIVE Plan: Pro Plus Run ID: 📒 Files selected for processing (3)
WalkthroughThe PR updates three benchmark JIT-statistics baselines. The changes record revised compiled bridges, guard failures, compiled loops, and aborted loops. No public declarations change. ChangesJIT statistics
Estimated code review effort: 1 (Trivial) | ~3 minutes Possibly related PRs
Poem
🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
🤖 Codex parity reviewStatic analysis of this diff vs the local RPython/PyPy sources (commit 6c3ac5c). The diff against the review base changes only generated jit-stats baselines (3 file(s)); nothing to review. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
pyre/check.pyis red onorigin/mainon all three OS runners (run30830792937, sha2c86af5). The same failures are present one commit earlier atc1e2c01(run30829934758), so they are not caused by the tip commit.What fails
exception_try_call_inlined_callee_raise—loops_compiled 3 -> 2exception_try_call_inlined_callee_raise—loops_compiled 3 -> 2exc_in_loop_divzero_continue—guard_failures 360 -> 603All three reproduce locally and are byte-stable across repeated runs, so none of them is runner noise.
Why the recorded values are wrong
exception_try_call_inlined_callee_raise— the recordedloops_compiled=3is a value no runner produces. The cranelift baseline wasloops_compiled=2, guard_failures=201until #1003 raised it to3 / 402; every platform, and every local run, reads2 / 201. This commit returns that file to the exact blob #1003 replaced. The wasm baseline for the same fixture gained itsloops_compiledfield in #1009, recorded as3, and CI has never observed that value — the field's first CI run (30820619419, shaa031486) already failed on it. Both files now hold the same content, which the two backends measure independently.loops_compiledis the sole member ofJITSTATS_FALL_FIELDSand fails on any fall, so lowering it deserves the scrutiny: this is not a loop that stopped being compiled, it is a number that was never reproduced.exc_in_loop_divzero_continue— this one is a real change, and the numbers say it is an improvement rather than a regression:#960 compiles bridges for this fixture where none were compiled before. The new bridges carry their own guards, so
guard_failuresrises alongside them.loops_compiledis unchanged, so no loop was lost. The gate flagged the rise because aguard_failuresrise is a regression by default; here it is the cost of bridges that now exist.Verification
python pyre/check.py --synthetic-only, run per backend on this branch:Individual fixtures were also measured directly against
pyre-craneliftandpyre-wasm-runner(3–5 runs each, identical every time) before the baselines were written, rather than only through check.py's snapshot path.Not addressed here
The
CPython suite (gate)job is also red:test.test_asyncio: PASS -> TIMEOUT. It starts at #1003 and is a behaviour change, not a recorded value, so raising the timeout would hide it. It needs its own investigation and is left untouched by this PR — main stays red on that job after this merges, while the threepyre/check.pyjobs go green.— authored by Claude
🤖 Generated with Claude Code
Summary by CodeRabbit