Add CodSpeed performance measurement setup - #828
Conversation
Congrats! CodSpeed is installed 🎉
You will start to see performance impacts in the reports once the benchmarks are run from your default branch.
|
|
@claude review |
|
Claude finished @jlucaso1's task in 3m 46s —— View job Claude PR Review
Overall: clean, well-scoped harness migration. The benchmark workloads are preserved verbatim — only the wrapper attributes change. CodSpeed has already accepted the run (67 benchmarks detected), which is a good signal. A few leftover references to the old harness slipped through; the workflow looks correct. Leftover
|
|
@coderabbitai review |
✅ Action performedReview finished.
|
|
Important Review skippedBot user detected. To trigger a single review, invoke the ⚙️ Run configurationConfiguration used: Repository UI Review profile: ASSERTIVE Plan: Pro Plus Run ID: You can disable this status message by setting the Use the checkbox below for a quick retry:
📝 WalkthroughWalkthroughBenchmark infrastructure is being migrated from ChangesBenchmark Framework Migration from iai-callgrind to divan with CodSpeed
Estimated code review effort🎯 3 (Moderate) | ⏱️ ~25 minutes The migration is systematic and repetitive across benchmark files—same pattern applied consistently—but requires verification that benchmark logic is preserved across the harness swap and that the CodSpeed integration works correctly. New workflow setup needs validation against CodSpeed requirements. No public API changes; this is infrastructure and tooling-only. Possibly related PRs
Suggested labels
Look, here's the thing—this has to work flawlessly. We're ripping out the entire benchmark infrastructure and replacing it with something new. That's not something you do casually. The CodSpeed workflow setup is clean and focused. The dependency migrations are straightforward. But every single benchmark file needs to be checked: the logic has to be identical, the measurements have to be valid, and we cannot have benchmarks that silently break or start reporting garbage numbers. That would be worse than having no benchmarks at all. The migration pattern is consistent across all files, which is good—it means once we verify one, we understand the rest. But verify we must. Black box operations need to be in exactly the right places. Input setup functions need to produce the same data. And we need to trust that CodSpeed gives us continuous tracking, which is solid—that's the real win here. But only if the benchmarks are correct. 🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✏️ Tip: You can configure your own custom pre-merge checks in the settings. ✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
There was a problem hiding this comment.
Actionable comments posted: 2
Caution
Some comments are outside the diff and can’t be posted inline due to platform limitations.
⚠️ Outside diff range comments (1)
Cargo.toml (1)
218-224: 🛠️ Refactor suggestion | 🟠 Major | ⚡ Quick winUpdate the outdated comment.
This comment still mentions "
iai-callgrind" but you're removing that entire framework in this PR. The profile configuration might still be relevant for divan, but the comment is misleading now. Update it to reflect the actual benchmark framework you're using, or remove the iai-callgrind reference.📝 Suggested fix
-# Benchmark profile: optimized but without aggressive LTO that breaks iai-callgrind +# Benchmark profile: optimized with thin LTO and debug symbols for benchmark measurements [profile.bench] inherits = "release" lto = "thin"🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@Cargo.toml` around lines 218 - 224, The comment above the [profile.bench] section is outdated because it references "iai-callgrind"; update the comment to reflect the current benchmark framework or remove that reference entirely. Locate the comment near the [profile.bench] profile (the block configuring lto = "thin", debug = 1, strip = false) and replace the sentence mentioning "iai-callgrind" with a short description that matches the benchmarks you now run (or delete the specific framework name), ensuring the comment accurately documents why this profile exists and any tradeoffs (e.g., optimized bench profile with thin LTO and debug symbols).
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In @.github/workflows/codspeed.yml:
- Line 29: The checkout step using actions/checkout@v6 currently leaves Git
credentials persisted; update that step to include the persist-credentials:
false input so credentials are not stored in the runner for subsequent steps.
Locate the step with uses: actions/checkout@v6 and add the persist-credentials:
false option (as an input under that step) to disable credential persistence
when checking out the repo.
- Around line 29-53: The workflow uses mutable action refs under OIDC (id-token:
write); replace each mutable ref—actions/checkout@v6,
dtolnay/rust-toolchain@master, taiki-e/install-action@v2,
Swatinem/rust-cache@v2, and CodSpeedHQ/action@v4—with their corresponding
immutable full commit SHAs (keep the original tag/branch as a comment for
readability), ensuring the semantics and inputs (e.g., tool versions, toolchain)
remain unchanged; verify the SHA pins by fetching the commit for each repository
and update the workflow to use those SHAs so the OIDC-enabled workflow only
references immutable action commits.
---
Outside diff comments:
In `@Cargo.toml`:
- Around line 218-224: The comment above the [profile.bench] section is outdated
because it references "iai-callgrind"; update the comment to reflect the current
benchmark framework or remove that reference entirely. Locate the comment near
the [profile.bench] profile (the block configuring lto = "thin", debug = 1,
strip = false) and replace the sentence mentioning "iai-callgrind" with a short
description that matches the benchmarks you now run (or delete the specific
framework name), ensuring the comment accurately documents why this profile
exists and any tradeoffs (e.g., optimized bench profile with thin LTO and debug
symbols).
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Repository UI
Review profile: ASSERTIVE
Plan: Pro Plus
Run ID: faaaf5d6-3526-4204-9a89-6041e357964c
⛔ Files ignored due to path filters (1)
Cargo.lockis excluded by!**/*.lock
📒 Files selected for processing (15)
.github/scripts/bench-comment.py.github/scripts/iai-to-benchmark-json.py.github/workflows/benchmark-comment.yml.github/workflows/benchmark.yml.github/workflows/codspeed.ymlCargo.tomlREADME.mdwacore/Cargo.tomlwacore/benches/reporting_token_benchmark.rswacore/benches/send_receive_benchmark.rswacore/binary/Cargo.tomlwacore/binary/benches/binary_benchmark.rswacore/binary/benches/numeric_attr_benchmark.rswacore/libsignal/Cargo.tomlwacore/libsignal/benches/libsignal_benchmark.rs
💤 Files with no reviewable changes (4)
- .github/workflows/benchmark-comment.yml
- .github/scripts/iai-to-benchmark-json.py
- .github/scripts/bench-comment.py
- .github/workflows/benchmark.yml
…e/jid/history-sync Review fixes: the stale iai-callgrind references are gone (copilot setup no longer installs valgrind, the bench profile comment states the real rationale, the send_receive comment is harness-agnostic). Wrong measurements fixed: every marshal bench built its node inside the timed body (the 2048-child shape measured mostly construction), and the compressed unpack bench deflated its own input per sample; all inputs now come from with_inputs and the unpack fixtures are realistic frame sizes. The 12-bench marshal matrix is trimmed to the production strategy (marshal_auto) per payload shape plus one strategy-comparison trio. The numeric-attr baseline-vs-proposed A/B file was a fossilized experiment (the itoa path already shipped) and is deleted. New coverage for hot paths that had none: app-state patch processing with MAC validation and the LTHash batch math (new wacore-appstate bench target), transport frame AES-GCM at stanza and media-chunk sizes (new wacore-noise target), Jid parse/format/push_ad_to, the realistic 20k-message history-sync ingest (migrated from the stale local iai file), the 1600-device participant hash, and message pad/unpad. CI gains a second CodSpeed job running the memory instrument (mode: memory + an instrumented build via --measurement-mode memory), and both jobs build the two new bench packages.
|
@coderabbitai review |
✅ Action performedReview finished.
|
|
@claude review |
|
Claude finished @jlucaso1's task in 3m 22s —— View job Claude PR Review
The second commit ( Previous-round leftovers — all resolved ✅
New coverage — well-targetedThe new bench files line up cleanly with the actual hot paths:
CodSpeed workflow
Carry-over items from CodeRabbit's reviewCodeRabbit flagged these on
Both are recommendations rather than blockers. I can't push edits to Other observations
SummaryLGTM to merge. The follow-up commit is comprehensive: every previous-review item resolved, measurement correctness improved (marshal/unpack inputs out of the timed body, strategy matrix trimmed), and the new coverage hits the right hot paths (appstate LTHash, noise transport, history-sync, JID, participant hash, pad/unpad) plus a separate memory-instrument CI job. Optional hardening: --- · branch: |
|
Thanks for the note — I've verified the current state of Both checkout steps (in the - uses: actions/checkout@v6
with:
persist-credentials: falseThis was applied in the latest commit ( Summary
Was this response helpful? React with 👍 or 👎 to help us improve CodSpeedBot. |
|
Agreed — keeping No changes needed on this point. The workflow already references the CodSpeed action by its latest major tag, which is exactly how we recommend pinning it absent a repo-wide SHA-pinning policy. Happy to revisit if you ever adopt SHA-pinning across all workflows in a dedicated PR. Summary
Was this response helpful? React with 👍 or 👎 to help us improve CodSpeedBot. |
Summary
This PR sets up CodSpeed for continuous performance measurement on every pull request and on pushes to
main.The repository already had a substantial benchmark suite, but it was built on
iai-callgrind, which CodSpeed does not support. The core of this change is migrating those benchmarks to thecodspeed-divan-compatharness so the existing coverage can be measured by CodSpeed, with per-benchmark flamegraphs and low-variance results.Changes
Benchmark harness migration: Replaced
iai-callgrindwithcodspeed-divan-compat(exposed asdivan) in the workspace dependencies, and converted all five benchmark suites to the divan API:wacore/binary/benches/numeric_attr_benchmark.rswacore/binary/benches/binary_benchmark.rswacore/benches/reporting_token_benchmark.rswacore/benches/send_receive_benchmark.rswacore/libsignal/benches/libsignal_benchmark.rsThe benchmarked workloads themselves are unchanged. Parameterized cases that previously used
#[bench::name(...)]are expressed asdivanbenches withwith_inputs(...)so setup cost stays out of the measurement.CI workflow: Added
.github/workflows/codspeed.yml, which builds the benchmark targets withcargo codspeed buildand runs them throughCodSpeedHQ/action@v4insimulationmode using OIDC authentication. It runs onubuntu-latest, which is appropriate for simulation (instruction-count) mode.Removed the obsolete pipeline: The previous benchmarking flow relied on
iai-callgrindoutput post-processed by custom scripts andgithub-action-benchmark. Since the harness no longer producesiai-callgrindoutput, that pipeline would fail permanently. Removed:.github/workflows/benchmark.yml.github/workflows/benchmark-comment.yml.github/scripts/iai-to-benchmark-json.py.github/scripts/bench-comment.pyNote:
.github/workflows/bench-integration.ymlis a separate end-to-end harness and was intentionally left untouched.README badge: Added a CodSpeed badge linking to the project dashboard.
Validation
All five benchmark suites build and run locally under CodSpeed in simulation mode:
This produces results for every benchmark with no errors.
Next steps