Add CodSpeed continuous performance benchmarks - #54
Conversation
Congrats! CodSpeed is installed 🎉
You will start to see performance impacts in the reports once the benchmarks are run from your default branch.
|
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: af7b6399cc
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
| let mut frame = PyFrame::new_with_context(code, execution_context) | ||
| .expect("benchmark frame creation must succeed"); | ||
|
|
||
| let _ = eval_with_jit(&mut frame); |
There was a problem hiding this comment.
Handle the eval_with_jit result instead of discarding it: if a benchmarked script starts raising (for example after a regression in bytecode execution or JIT tracing), this line treats the run as successful and CodSpeed will time a fast failing path, producing misleading performance data rather than surfacing a broken workload. Benchmark runs should fail immediately on Err (e.g., with expect) so regressions do not silently corrupt measurements.
Useful? React with 👍 / 👎.
The test was named `jit_factorial` while its body called only `interp::interpret` -- no `jit.run`, no `mainloop`. It could not fail for any JIT reason, so the name asserted a tier the body never touched. Renamed to what it actually covers, and the absent JIT twin is recorded as deliberate rather than left to be rediscovered: tiny2 is one of #32's no-greens crates and OP_LOOP_END is an abort stub (#45), so a JIT arm added today could not compile anything -- a gate over a tier that cannot run, which is the cost #48 measured. The note says to add the twin once greens is declared and `]` lowers, and to route it through `run_locked` so it takes the probe lock added in #54. Sole reference in the tree; no callers to update. tiny2 15/15, fmt clean. Assisted-by: Claude
Summary
This PR integrates CodSpeed for continuous performance monitoring of the pyre interpreter and JIT pipeline. Every push to
mainand every pull request now runs reproducible benchmarks and reports performance changes automatically.Changes
New benchmark crate:
pyre-benchA new
pyre/pyre-benchcrate usescodspeed-criterion-compatto benchmark 6 Python workloads through the full interpreter + JIT pipeline:Each benchmark compiles a Python script, creates an interpreter frame, and evaluates it through
eval_with_jit, mirroring the real execution path. Iteration counts are scaled down from the existingpyre/bench/scripts to keep CI times reasonable while still exercising the same code paths.CI workflow:
.github/workflows/codspeed.ymlmain, pull requests, andworkflow_dispatch(for CodSpeed backtesting)majit/andpyre/path changes to avoid unnecessary runsREADME badge
Added a CodSpeed badge to the project README alongside the existing Discord badge.
Note on runners
This repository belongs to a personal GitHub account, so
codspeed-macrorunners are not available. The workflow usesubuntu-latestrunners instead. This means performance variance will be higher than with dedicated bare-metal runners. If this repository is moved to a GitHub organization in the future, switching to macro runners is recommended for more stable measurements. See CodSpeed walltime docs for details.Next steps
main