Skip to content

Commit b4a8f78

Browse files
authored
fix(metrics): widen lean_block_building_time_seconds buckets past 1s (lambdaclass#574)
## 🗒️ Description / Motivation `lean_block_building_time_seconds` used the leanMetrics bucket set, whose top finite bound is 1s. Block builds on our devnets routinely take 2-3s, so in practice every sample landed in the `+Inf` bucket. When a quantile falls in `+Inf`, `histogram_quantile` has no finite bucket left to interpolate in and returns the upper bound of the last finite one. The p99 panel was reporting a flat 1s line no matter how long builds actually took, so the metric could not answer the question it exists to answer. This widens the range to 0.1s - 8s, matching `lean_block_proposal_attestation_build_phase_seconds`, whose phases this metric encloses. The sibling `lean_block_building_payload_aggregation_time_seconds` already reaches 4s, so a wider range on the enclosing timer is consistent with what's already there. ## What Changed - `crates/blockchain/src/metrics.rs` — buckets `0.01…1` → `0.1, 0.25, 0.5, 0.75, 1, 2, 4, 8`, plus a comment recording why this deviates from the spec set. - `docs/metrics.md` — updated the Block Production table row and added a note below the table explaining the deviation. ## Correctness / Behavior Guarantees No behavior change: this is bucket boundaries on a histogram, nothing on the block production path. **Tradeoff.** Starting at 0.1 drops the `0.01 / 0.025 / 0.05` boundaries. Attestation-free blocks skip the prover and build in single-digit milliseconds, and those now all fall into the first bucket. `lean_block_proposal_attestation_build_phase_seconds` still resolves that end down to 0.001, and the sub-100ms region isn't the one we need to watch, so this seemed like the right side to give up. **Cross-client impact: none.** Prometheus stores each bucket as an independent series, so other clients' exposition is untouched, and every boundary that was shared before (`0.1` through `1`) is still present here. Divergent bucket sets only corrupt a query that does `sum by (le)` *across* clients; I checked the leanMetrics dashboards, and the panel for this metric queries per-instance (`rate(..._bucket{network=~, job=~, instance=~})`) with no `sum by (le)`, so each histogram stays internally consistent. `rate(_sum)/rate(_count)` is unaffected either way. This does put us out of step with the leanMetrics spec, which is why both the code and the docs carry a note: without one, this reads like drift and gets "corrected" back to a set that can't measure the thing. Worth proposing the wider range upstream so the other clients get a usable panel too. ## Tests Added / Run No new tests: bucket boundaries have no assertions to make against them. - `make fmt` — clean - `make lint` — clean - `make test` — 30/30 test binaries pass, 0 failures (fork-choice, STF, signature, and SSZ spec tests included) ## Related Issues / PRs - Related to lambdaclass#566 (previous metrics work) ## ✅ Verification Checklist - [x] Ran `make fmt` — clean - [x] Ran `make lint` (clippy with `-D warnings`) — clean - [x] Ran `make test` (`cargo test --workspace --profile release-fast`) — all passing
1 parent 4209de9 commit b4a8f78

2 files changed

Lines changed: 11 additions & 2 deletions

File tree

crates/blockchain/src/metrics.rs

Lines changed: 4 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -460,12 +460,15 @@ static LEAN_BLOCK_BUILDING_PAYLOAD_AGGREGATION_TIME_SECONDS: std::sync::LazyLock
460460
.unwrap()
461461
});
462462

463+
// Widened past the leanMetrics bucket set: block builds regularly exceed its top bound,
464+
// which collapsed every sample into `+Inf` and pinned the reported quantiles to that
465+
// bound. The range mirrors the phase timings this metric encloses.
463466
static LEAN_BLOCK_BUILDING_TIME_SECONDS: std::sync::LazyLock<Histogram> =
464467
std::sync::LazyLock::new(|| {
465468
register_histogram!(
466469
"lean_block_building_time_seconds",
467470
"Time taken to build a block",
468-
vec![0.01, 0.025, 0.05, 0.1, 0.25, 0.5, 0.75, 1.0]
471+
vec![0.1, 0.25, 0.5, 0.75, 1.0, 2.0, 4.0, 8.0]
469472
)
470473
.unwrap()
471474
});

docs/metrics.md

Lines changed: 7 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -36,7 +36,7 @@ The exposed metrics follow [the leanMetrics specification](https://github.com/le
3636
|--------|-------|-------|-------------------------|--------|---------|-----------|
3737
| `lean_block_aggregated_payloads` | Histogram | Number of `aggregated_payloads` in a block | On block production | | 1, 2, 4, 8, 16, 32, 64, 128 ||
3838
| `lean_block_building_payload_aggregation_time_seconds` | Histogram | Time taken to build `aggregated_payloads` during block building | On block production | | 0.1, 0.25, 0.5, 0.75, 1, 2, 3, 4 ||
39-
| `lean_block_building_time_seconds` | Histogram | Time taken to build a block | On block production | | 0.01, 0.025, 0.05, 0.1, 0.25, 0.5, 0.75, 1 ||
39+
| `lean_block_building_time_seconds` | Histogram | Time taken to build a block | On block production | | 0.1, 0.25, 0.5, 0.75, 1, 2, 4, 8 ||
4040
| `lean_block_building_success_total` | Counter | Successful block builds | On block production | | ||
4141
| `lean_block_building_failures_total` | Counter | Failed block builds (error building the block, signing the block root, or processing it locally) | On block production failure | | ||
4242
| `lean_block_proposal_attestation_build_phase_seconds` | Histogram | Phase-level time in block-proposal attestation selection | On block production | phase=select_payloads,compact,stf_simulate | 0.001, 0.005, 0.01, 0.025, 0.05, 0.1, 0.25, 0.5, 1, 2, 4, 8 ||
@@ -45,6 +45,12 @@ The exposed metrics follow [the leanMetrics specification](https://github.com/le
4545
| `lean_block_proposal_attestation_data_selected` | Histogram | Distinct `AttestationData` entries in the proposal block body | On block production | | 0, 1, 2, 4, 8, 16, 32 ||
4646
| `lean_block_proposal_aggregates_selected` | Histogram | Aggregated signature proofs in the proposal result after compaction | On block production | | 0, 1, 2, 4, 8, 16, 32, 64, 128 ||
4747

48+
> `lean_block_building_time_seconds` intentionally deviates from the leanMetrics bucket
49+
> set, which tops out at 1s. Real builds on our devnets routinely run past that, so every
50+
> sample landed in `+Inf` and `histogram_quantile` reported a flat 1s ceiling. The range
51+
> now covers the same span as the `lean_block_proposal_attestation_build_phase_seconds`
52+
> phases it contains.
53+
4854
## Fork-Choice Metrics
4955

5056
| Name | Type | Usage | Sample collection event | Labels | Buckets | Supported |

0 commit comments

Comments
 (0)