Skip to content

Commit de60642

Browse files
committed
fix(metrics): widen lean_block_building_time_seconds buckets past 1s
The leanMetrics bucket set for this histogram tops out at 1s, but block builds on our devnets routinely take 2-3s. Every sample therefore landed in `+Inf`, and `histogram_quantile` had no finite bucket left to interpolate in, so it returned the upper bound of the last finite bucket. The panel read as a flat 1s line regardless of how long builds actually took. Buckets now span 0.1s to 8s, the same range as `lean_block_proposal_attestation_build_phase_seconds`, whose phases this metric encloses. The tradeoff is sub-100ms resolution: attestation-free blocks skip the prover and build in single-digit milliseconds, and those now all collapse into the first bucket. The phase histogram still resolves that end, and it is not the range we need to watch. This deviates from the leanMetrics spec, so both the histogram and docs/metrics.md carry a note explaining why, to keep it from being "corrected" back to a set that cannot measure the thing. Divergent buckets do not affect other clients: Prometheus stores each bucket as its own series, and every boundary that was shared before (0.1 through 1) is still present. The leanMetrics dashboard panel for this metric queries per-instance rather than `sum by (le)` across clients, so each histogram stays internally consistent.
1 parent f022277 commit de60642

2 files changed

Lines changed: 11 additions & 2 deletions

File tree

crates/blockchain/src/metrics.rs

Lines changed: 4 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -460,12 +460,15 @@ static LEAN_BLOCK_BUILDING_PAYLOAD_AGGREGATION_TIME_SECONDS: std::sync::LazyLock
460460
.unwrap()
461461
});
462462

463+
// Widened past the leanMetrics bucket set: block builds regularly exceed its top bound,
464+
// which collapsed every sample into `+Inf` and pinned the reported quantiles to that
465+
// bound. The range mirrors the phase timings this metric encloses.
463466
static LEAN_BLOCK_BUILDING_TIME_SECONDS: std::sync::LazyLock<Histogram> =
464467
std::sync::LazyLock::new(|| {
465468
register_histogram!(
466469
"lean_block_building_time_seconds",
467470
"Time taken to build a block",
468-
vec![0.01, 0.025, 0.05, 0.1, 0.25, 0.5, 0.75, 1.0]
471+
vec![0.1, 0.25, 0.5, 0.75, 1.0, 2.0, 4.0, 8.0]
469472
)
470473
.unwrap()
471474
});

docs/metrics.md

Lines changed: 7 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -36,7 +36,7 @@ The exposed metrics follow [the leanMetrics specification](https://github.com/le
3636
|--------|-------|-------|-------------------------|--------|---------|-----------|
3737
| `lean_block_aggregated_payloads` | Histogram | Number of `aggregated_payloads` in a block | On block production | | 1, 2, 4, 8, 16, 32, 64, 128 ||
3838
| `lean_block_building_payload_aggregation_time_seconds` | Histogram | Time taken to build `aggregated_payloads` during block building | On block production | | 0.1, 0.25, 0.5, 0.75, 1, 2, 3, 4 ||
39-
| `lean_block_building_time_seconds` | Histogram | Time taken to build a block | On block production | | 0.01, 0.025, 0.05, 0.1, 0.25, 0.5, 0.75, 1 ||
39+
| `lean_block_building_time_seconds` | Histogram | Time taken to build a block | On block production | | 0.1, 0.25, 0.5, 0.75, 1, 2, 4, 8 ||
4040
| `lean_block_building_success_total` | Counter | Successful block builds | On block production | | ||
4141
| `lean_block_building_failures_total` | Counter | Failed block builds (error building the block, signing the block root, or processing it locally) | On block production failure | | ||
4242
| `lean_block_proposal_attestation_build_phase_seconds` | Histogram | Phase-level time in block-proposal attestation selection | On block production | phase=select_payloads,compact,stf_simulate | 0.001, 0.005, 0.01, 0.025, 0.05, 0.1, 0.25, 0.5, 1, 2, 4, 8 ||
@@ -45,6 +45,12 @@ The exposed metrics follow [the leanMetrics specification](https://github.com/le
4545
| `lean_block_proposal_attestation_data_selected` | Histogram | Distinct `AttestationData` entries in the proposal block body | On block production | | 0, 1, 2, 4, 8, 16, 32 ||
4646
| `lean_block_proposal_aggregates_selected` | Histogram | Aggregated signature proofs in the proposal result after compaction | On block production | | 0, 1, 2, 4, 8, 16, 32, 64, 128 ||
4747

48+
> `lean_block_building_time_seconds` intentionally deviates from the leanMetrics bucket
49+
> set, which tops out at 1s. Real builds on our devnets routinely run past that, so every
50+
> sample landed in `+Inf` and `histogram_quantile` reported a flat 1s ceiling. The range
51+
> now covers the same span as the `lean_block_proposal_attestation_build_phase_seconds`
52+
> phases it contains.
53+
4854
## Fork-Choice Metrics
4955

5056
| Name | Type | Usage | Sample collection event | Labels | Buckets | Supported |

0 commit comments

Comments
 (0)