feat(drive-abci): debug-only per-block phase timing

[PastaPastaPasta]

Issue being fixed or feature implemented

There was no way to see where a block's time goes inside drive-abci. ProcessProposal logged one elapsed_time_ms — truncated to whole milliseconds — and FinalizeBlock logged nothing at all, so more than half the per-block cost was unattributed.

That gap hid two costs that scale with chain history and together accounted for most of a mainnet sync:

• an unbounded withdrawal-document query, 4.8 ms/block by height 200,000 (perf(drive-abci): don't read every withdrawal document to build a debug log line #4569)
• GroveDB checkpoint creation during replay, 15.1 ms/block from height 318,704 (perf(drive-abci): don't create GroveDB checkpoints while replaying history #4570)

Neither is visible without per-phase numbers. Both were found with this.

What was done?

In debug builds, a Laps value times successive phases of block execution and merges them into process-wide totals on drop. perf::end_block reports the means every DRIVE_BLOCK_PERF_EVERY blocks (default 500) as a single log line.

Two design points worth noting:

• Compiled out of standard release builds. The perf module and every timing call site are guarded by #[cfg(debug_assertions)]. Release builds contain no phase timers, environment reads, totals mutex, or report logging from this feature, even when DRIVE_BLOCK_PERF=1 is set. In builds with debug assertions enabled, that environment switch still defaults to off and is read once through OnceLock<bool>; when off, Laps::new allocates nothing and every lap returns immediately.
• Accumulated, not logged per block. An earlier version emitted a line per block and the JSON formatting landed inside the spans being measured, inflating exactly the phases under investigation. Means are reported periodically instead.

The mean is over blocks rather than over samples, so a phase that only runs on some blocks shows its share of the per-block cost rather than its cost when it fires. Sample counts are reported alongside, which is how the fire rate of a phase becomes visible.

fb_proposal in the finalize handler wraps the whole of finalize_block_proposal, so it is the sum of the fbp_* phases; add up laps from one level only.

Phases covered: the block-proposal path (epoch info, block-cache clear, state clone, core info, chain lock, withdrawals, DAO events, state transitions, fees, root hash, validator set) and the finalize path (proposal validation, commit signature verification, drive cache, state cache, commit, checkpoint).

Example output:

block perf  height=195000 blocks=500
  core_info=1220/331 fbp_verify_commit=567/500 chainlock=456/331
  fb_commit=354/500 wd_status=335/331 dao=187/500 state_clone=154/500 ...

Each term is name=mean_µs_per_block/blocks_it_ran_on. core_info, chainlock and wd_status only run on blocks that advance the core height, which is why their sample counts are below 500; phases that run on every block show /500.

How Has This Been Tested?

Used throughout a full mainnet replay, genesis to 424,981, and for every A/B measurement behind #4569, #4570, #4571 and #4572.

cargo test -p drive-abci --lib — 2,770 passed. The follow-up adds unit tests for the accumulation, mean-over-blocks and interval reset logic in perf.rs.

Release-exclusion validation:

• cargo build -p drive-abci --release --locked passed.
• cargo test -p drive-abci --lib --locked -- perf:: passed: 5 tests.
• The release executable has no DRIVE_BLOCK_PERF strings, phase/report markers, or drive_abci::perf symbols; the debug test executable retains the instrumentation as a positive control.
• A compile probe referencing drive_abci::perf::Laps against the release library fails because the module is configured out.
• cargo clippy -p drive-abci --lib --locked -- -D warnings, formatting, and git diff --check passed.

Breaking Changes

None for standard release builds: all phase-timing instrumentation is excluded at compile time. Debug builds require DRIVE_BLOCK_PERF=1 to enable it.

Checklist:

• I have performed a self-review of my own code
• I have commented my code, particularly in hard-to-understand areas
• I have added or updated relevant unit/integration/functional/e2e tests
• I have made corresponding changes to the documentation

For repository code-owners and collaborators only

• I have assigned this pull request to a milestone

🤖 Generated with Claude Code

[coderabbitai]

Warning

Review limit reached

Next included review available in 45 minutes.

Check out review usage here.

View limit details

Limit details: You’ve used the included review currently available.

You've used all free OSS reviews for now. Wait for the free limit to reset to keep reviewing this public repository.

Learn how review limits work.

Review configuration:

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Advanced

Run ID: 6dc20c94-bc5d-40b0-9d19-495d293b6dfd

📥 Commits

Reviewing files that changed from the base of the PR and between c0e9a86 and 2e7fb4d.

📒 Files selected for processing (6)

• packages/rs-drive-abci/src/abci/handler/finalize_block.rs
• packages/rs-drive-abci/src/execution/engine/finalize_block_proposal/v0/mod.rs
• packages/rs-drive-abci/src/execution/engine/run_block_proposal/mod.rs
• packages/rs-drive-abci/src/execution/engine/run_block_proposal/v0/mod.rs
• packages/rs-drive-abci/src/lib.rs
• packages/rs-drive-abci/src/perf.rs

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[thepastaclaw]

🕓 Queued for automated review — 44th in line, estimated start in ~63 h (commit 2e7fb4d)
Estimated review time once started: ~2.9 h (two-phase automated review; median of recent runs).

• Request priority review — tick this box and the review moves to the front of the queue.

[codecov]

Codecov Report

❌ Patch coverage is 49.84520% with 162 lines in your changes missing coverage. Please review.
✅ Project coverage is 85.56%. Comparing base (17a2962) to head (2e7fb4d).
⚠️ Report is 71 commits behind head on v4.2-dev.

Files with missing lines	Patch %	Lines
packages/rs-drive-abci/src/perf.rs	55.61%	87 Missing ⚠️
.../src/execution/engine/run_block_proposal/v0/mod.rs	40.62%	38 Missing ⚠️
...execution/engine/finalize_block_proposal/v0/mod.rs	43.33%	17 Missing ⚠️
...s/rs-drive-abci/src/abci/handler/finalize_block.rs	37.50%	15 Missing ⚠️
...bci/src/execution/engine/run_block_proposal/mod.rs	44.44%	5 Missing ⚠️

Additional details and impacted files

@@             Coverage Diff              @@
##           v4.2-dev    #4573      +/-   ##
============================================
- Coverage     87.57%   85.56%   -2.01%     
============================================
  Files          2748     2763      +15     
  Lines        357005   368816   +11811     
============================================
+ Hits         312647   315582    +2935     
- Misses        44358    53234    +8876     

Components	Coverage Δ
dpp	85.48% <ø> (-2.90%)	⬇️
drive	84.65% <ø> (-1.73%)	⬇️
drive-abci	88.49% <49.84%> (-1.40%)	⬇️
sdk	∅ <ø> (∅)
dapi-client	∅ <ø> (∅)
platform-version	∅ <ø> (∅)
platform-value	92.92% <ø> (ø)
platform-wallet	∅ <ø> (∅)
drive-proof-verifier	41.77% <ø> (-6.88%)	⬇️

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[thepastaclaw]

Final validation — GLM Flash + Sol

The opt-in timing instrumentation is consensus-inert and appropriately keeps logging outside measured spans, but several accounting details reduce the trustworthiness of its output: conditional phases report misleading sample counts, and protocol-change work remains unattributed. The aggregation lifecycle also lacks focused tests, and best-effort telemetry should not be able to panic block processing after mutex poisoning.

Source: reviewer 1: glm-5.3-flash (agent: phase1-reviewer, role: general); reviewer 2: glm-5.3-flash (agent: phase1-reviewer, role: security-auditor); reviewer 3: glm-5.3-flash (agent: phase1-reviewer, role: rust-quality); reviewer 4: gpt-5.6-sol (agent: phase2-reviewer, role: general); reviewer 5: gpt-5.6-sol (agent: phase2-reviewer, role: security-auditor); reviewer 6: gpt-5.6-sol (agent: phase2-reviewer, role: rust-quality); final verifier: gpt-5.6-sol (agent: sol-verifier, role: final-verifier)

Review provenance

• Phase 1 reviewers (GLM Flash): glm-5.3-flash — general (completed); agent phase1-reviewer, glm-5.3-flash — security-auditor (completed); agent phase1-reviewer, glm-5.3-flash — rust-quality (completed); agent phase1-reviewer
• Fresh verifier (Sol): gpt-5.6-sol — final-verifier; agent sol-verifier
• Phase 2 reviewers (Sol): gpt-5.6-sol — general (completed); agent phase2-reviewer, gpt-5.6-sol — security-auditor (completed); agent phase2-reviewer, gpt-5.6-sol — rust-quality (completed); agent phase2-reviewer

🟡 4 suggestion(s)

1 additional finding(s) omitted (not in diff).

🤖 Prompt for all review comments with AI agents

These findings are from an automated code review. Verify each finding against the current code and only fix it if needed.

In `packages/rs-drive-abci/src/abci/handler/finalize_block.rs`:
- [SUGGESTION] packages/rs-drive-abci/src/abci/handler/finalize_block.rs:108-113: Conditional phases report one sample even when skipped
  `fb_checkpoint` is sampled on every finalized block even though checkpoint creation only runs when `checkpoint_needed` is true. A window with one checkpoint and 499 ordinary blocks therefore reports `/500`, not `/1`, so the sample count cannot provide the advertised fire rate. The same issue affects `chainlock`, `wd_status`, and `fbp_wd_broadcast`, whose lap calls also sit outside their conditional branches. Add a timer operation that always advances the boundary but records a sample only when the corresponding work executed, and use it consistently for conditional phases.

In `packages/rs-drive-abci/src/execution/engine/run_block_proposal/mod.rs`:
- [SUGGESTION] packages/rs-drive-abci/src/execution/engine/run_block_proposal/mod.rs:86-149: Protocol-change migrations are omitted from the phase totals
  After recording `state_clone`, this function resolves the block platform version and may execute `perform_events_on_first_block_of_protocol_change`, then enters the versioned implementation without another outer lap. `Laps::drop` only merges recorded entries and does not record the elapsed tail, while the inner `Laps` starts after the migration. Protocol-activation work can therefore be substantial yet entirely absent from a replay report. Record this segment after the protocol-change event, preferably only when the migration executes so its sample count also reflects the actual fire rate.

In `packages/rs-drive-abci/src/perf.rs`:
- [SUGGESTION] packages/rs-drive-abci/src/perf.rs:26-160: No unit tests cover the new aggregation and reporting logic
  The module has no focused coverage for `Totals::add`, first-seen phase ordering, sum/sample accumulation, mean-over-block arithmetic, or clearing both phases and the block counter at an interval boundary. These calculations directly determine the performance measurements and can regress while still producing plausible output. The process-global `OnceLock` configuration makes environment-based tests order-dependent, but `Totals` can be tested directly and report formatting/reset can be extracted into a pure helper for deterministic unit tests.
- [SUGGESTION] packages/rs-drive-abci/src/perf.rs:96-127: Poisoned perf mutex can abort block processing
  All three mutex acquisitions call `expect("block perf totals poisoned")`, including the acquisition in `Drop for Laps`. If the mutex is ever poisoned, this diagnostics-only feature will panic during block processing; if the drop occurs during unwinding, the second panic aborts the process. The `expect` message also does not document an invariant proving poisoning impossible. Since these counters are best-effort telemetry and the inner value remains usable, recover consistently with `PoisonError::into_inner` at every lock site.

[PastaPastaPasta]

Review

Verdict: merge after changes. Inert unless the environment variable is set, so the risk is low; the changes are CI hygiene and two small correctness points in the instrument itself.

1. Correctness

The timing logic is right: Instant deltas accumulated in memory, merged into process totals on drop, reported as a mean over blocks. Two issues:

• Sample counts do not mean fire rate for conditional phases. fb_checkpoint, chainlock, wd_status and fbp_wd_broadcast call lap outside their if, so they record a near-zero sample on every block. The description says the sample count "is how the fire rate of a phase becomes visible"; for these four it is not. Add a lap_if(condition, name) that always advances the boundary and records only when the work ran.
• expect on a poisoned mutex inside Drop. If a block panics while the totals lock is held, the next Laps drop during unwinding panics again and aborts the process. This is telemetry; recover with unwrap_or_else(PoisonError::into_inner) at all three lock sites.

Also: fb_proposal wraps the whole of finalize_block_proposal, so it overlaps the fbp_* phases. That is fine as a nested total, but the module doc should say so, or a reader will add them up.

2. Clarity

The description explains the design choices well (accumulate instead of log per block, mean over blocks). The example output lists sps_serialize, which no lap in this PR produces; it came from a branch that also instrumented store_platform_state. Please make the example match the code.

3. Codebase standards

• CI is red on the title: bench is not an allowed type in .github/workflows/pr.yml. This is instrumentation, so feat(drive-abci): ... fits. Squash merges use the PR title, so the commit type follows.
• The codebase configures operators through PlatformConfig (envy), not raw std::env::var inside modules. For a diagnostic switch that must stay zero-cost and zero-touch I think the OnceLock read is defensible, but it is a deviation and the module doc should say why.
• Laps::on() and perf::value() have no callers. Remove them; whoever needs them can add them back with a use.
• No unit tests, which is why codecov/patch fails. Totals::add and the report formatting are pure and easy to test once the formatting is pulled out of end_block.

4. Importance and alternatives

This found #4569 and #4570, so it has paid for itself. The idiomatic alternative in this codebase is Prometheus histograms via crate::metrics, which would give per-phase distributions to any scraper. For a replay benchmark a single periodic log line is easier to read, and the two are not exclusive. I would merge this and consider histograms later.

5. Existing bot findings

All three thepastaclaw comments (conditional sample counts, poisoned mutex, tests) are valid and addressed above.

I will push: title to feat, lap_if, poison recovery, removal of dead API, tests, and a corrected example in the description.

---

🤖 Posted autonomously by Claude on behalf of pasta.

[PastaPastaPasta]

Pushed 19f82f4 and 9826308 on top of the original commit, and retitled the PR to feat(drive-abci): … since bench is not an allowed type; the title check is green.

• Laps::lap_if(ran, name) always moves the lap boundary and records a sample only when the work ran. fb_checkpoint, chainlock, wd_status and fbp_wd_broadcast use it, so their sample counts are now the number of blocks the work ran on. For chainlock the condition is core_chain_lock_update.is_some() && !known_from_us, because the verification only runs for a lock this node did not propose (my first version missed the second half; caught in review).
• A poisoned totals lock is recovered with PoisonError::into_inner at every site, so telemetry can never turn a panic into an abort during unwinding.
• Unused Laps::on() and perf::value() removed.
• Report formatting moved into Totals::report_line / Totals::end_block, with five unit tests (first-seen order, mean over blocks not samples, interval reset, empty report, lap_if skipped path). codecov/patch is green.
• Module doc says why this reads the environment instead of PlatformConfig, and that fb_proposal nests the fbp_* laps. end_block documents that an interval of zero reports every block.
• PR description: the example output now matches what the code prints (name=mean/samples) and no longer lists a phase this PR does not have.

cargo test -p drive-abci --lib -- perf::: 5 passed. fmt and clippy clean.

Ready for human review.

---

🤖 Posted autonomously by Claude on behalf of pasta.