Skip to content

Commit c2da08d

Browse files
committed
feat(cli): add offline block-building benchmark sub-command
`ethlambda benchmark synthetic` measures block building exactly as executed when the node proposes, against a reproducible synthetic workload, with no devnet required. "Optimize block building" (#465) is the top roadmap item, but the only observability today is Prometheus histograms on a live devnet: noisy, not reproducible, and unable to compare an optimization against a baseline. The harness drives the production proposer entry point (`produce_block_with_signatures`) over a seeded in-memory chain, seeding the pending pool per slot and letting the proposal tick promote it, as on a live node. Per-phase timings come from the existing `lean_block_proposal_attestation_build_phase_seconds` histogram: the per-label sample sums are deltaed between iterations, so attribution is exact and the hot path is untouched. Each iteration reports its block root, so a baseline-vs-optimized diff proves an optimization changed only speed and not attestation selection. `benchmark` is a second clap sub-command alongside `node`, so clap lists it in `--help` and names it in its own usage lines, and the harness arguments live in their own `Args` group rather than in CliOptions. `main` becomes synchronous and only the node path enters the tokio runtime: the benchmark is synchronous CPU-bound work and would otherwise park a worker thread for its whole run. Its logs go to stderr so the report owns stdout. NEW_PAYLOAD_CAP becomes public so the harness can reject a --proofs-per-data batch the pending pool would evict whole. `make bench` runs it. Reports one row per measured iteration. Aggregate statistics, build provenance and machine-readable output follow separately, as does the real-crypto workload — see docs/plans/block-building-benchmark.md for the milestones. Laid out to be read in one pass: `SyntheticCorpus` builds the chain and seeds the pool, `build_one_slot` is one slot's work end to end (seed, time the build, import), `PhaseTimer` turns two histogram readings into per-phase durations, `run_synthetic` is the loop over slots, and report.rs formats. Warmup and measured slots run the same code path; only whether the sample is kept differs.
1 parent d54044c commit c2da08d

11 files changed

Lines changed: 709 additions & 23 deletions

File tree

Cargo.lock

Lines changed: 1 addition & 0 deletions
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

Makefile

Lines changed: 6 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,4 +1,4 @@
1-
.PHONY: help fmt lint docker-build shadow-build shadow-docker-build run-devnet test docs docs-deps docs-serve
1+
.PHONY: help fmt lint bench docker-build shadow-build shadow-docker-build run-devnet test docs docs-deps docs-serve
22

33
help: ## 📚 Show help for each of the Makefile recipes
44
@grep -E '^[a-zA-Z0-9_-]+:.*?## .*$$' $(MAKEFILE_LIST) | sort | awk 'BEGIN {FS = ":.*?## "}; {printf "\033[36m%-30s\033[0m %s\n", $$1, $$2}'
@@ -14,6 +14,11 @@ test: leanSpec/fixtures ## 🧪 Run all tests
1414
# signature verification/aggregation, without paying for LTO on every rebuild
1515
cargo test --workspace --profile release-fast
1616

17+
BENCH_ARGS ?= synthetic --mock-crypto
18+
19+
bench: ## 🏁 Benchmark block building offline (override BENCH_ARGS to customize)
20+
cargo run --release --bin ethlambda -- benchmark $(BENCH_ARGS)
21+
1722
GIT_COMMIT=$(shell git rev-parse HEAD)
1823
GIT_BRANCH=$(shell git rev-parse --abbrev-ref HEAD)
1924
DOCKER_TAG?=local

bin/ethlambda/Cargo.toml

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -21,6 +21,7 @@ shadow-integration = ["ethlambda-crypto/shadow-integration"]
2121
[dependencies]
2222
ethlambda-blockchain.workspace = true
2323
ethlambda-crypto.workspace = true
24+
ethlambda-metrics.workspace = true
2425
ethlambda-network-api.workspace = true
2526
ethlambda-p2p.workspace = true
2627
ethlambda-types.workspace = true
Lines changed: 163 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,163 @@
1+
//! Synthetic benchmark corpus: deterministic validators, a genesis store, and
2+
//! per-slot attestation-pool seeding.
3+
4+
use std::sync::Arc;
5+
6+
use ethlambda_blockchain::store::produce_attestation_data;
7+
use ethlambda_storage::{Store, backend::InMemoryBackend};
8+
use ethlambda_types::{
9+
attestation::{AggregationBits, HashedAttestationData},
10+
block::SingleMessageAggregate,
11+
state::{State, Validator, ValidatorPubkeyBytes},
12+
};
13+
14+
/// Fixed genesis time for synthetic runs. The harness derives every tick
15+
/// timestamp from slot numbers relative to this value and never reads the wall
16+
/// clock, so runs are reproducible at any time of day.
17+
const GENESIS_TIME: u64 = 1_700_000_000;
18+
19+
pub(crate) struct SyntheticCorpus {
20+
num_validators: u64,
21+
proofs_per_data: u64,
22+
}
23+
24+
impl SyntheticCorpus {
25+
pub(crate) fn new(num_validators: u64, proofs_per_data: u64) -> Self {
26+
Self {
27+
num_validators,
28+
proofs_per_data,
29+
}
30+
}
31+
32+
/// Build a genesis store over an in-memory backend with `num_validators`
33+
/// seed-derived validators.
34+
///
35+
/// Pubkeys are deterministic placeholder bytes: in mock-crypto mode no code
36+
/// path decodes them (signature verification is skipped and best-proof
37+
/// compaction never resolves pubkeys).
38+
pub(crate) fn genesis_store(&self, seed: u64) -> Store {
39+
let mut rng_state = seed;
40+
let validators = (0..self.num_validators)
41+
.map(|index| Validator {
42+
attestation_pubkey: synthetic_pubkey(&mut rng_state),
43+
proposal_pubkey: synthetic_pubkey(&mut rng_state),
44+
index,
45+
})
46+
.collect();
47+
let genesis_state = State::from_genesis(GENESIS_TIME, validators);
48+
Store::from_anchor_state(Arc::new(InMemoryBackend::new()), genesis_state)
49+
}
50+
51+
/// Seed the pending ("new") pool with the full validator set's attestations
52+
/// for `attestation_slot`, split into `proofs_per_data` disjoint aggregates.
53+
///
54+
/// Mirrors what committee aggregators gossip during a slot: several
55+
/// aggregates for the same `AttestationData`, each covering a validator
56+
/// subset. The proposal tick then promotes them to the known pool, exactly
57+
/// as on a live node. Entries are inserted in a fixed order because pool
58+
/// insertion order pins within-entry proof choice during selection.
59+
///
60+
/// Returns the total number of pool entries the next build will see, across
61+
/// both pools.
62+
pub(crate) fn seed_pool(
63+
&self,
64+
store: &mut Store,
65+
attestation_slot: u64,
66+
) -> eyre::Result<usize> {
67+
let data = produce_attestation_data(store, attestation_slot);
68+
let entries = participant_groups(self.num_validators, self.proofs_per_data)
69+
.into_iter()
70+
.map(|participants| {
71+
(
72+
HashedAttestationData::new(data.clone()),
73+
SingleMessageAggregate::empty(participants),
74+
)
75+
})
76+
.collect();
77+
store.insert_new_aggregated_payloads_batch(entries);
78+
79+
// The pending pool evicts whole data-root entries FIFO once its proof
80+
// cap is exceeded, so an over-cap batch seeds nothing and every
81+
// measured block would come out empty.
82+
let pending = store.new_aggregated_payloads_count();
83+
eyre::ensure!(
84+
pending > 0,
85+
"attestations seeded for slot {attestation_slot} were evicted from the pending pool; \
86+
the measured workload would not match the requested parameters"
87+
);
88+
Ok(pending + store.known_aggregated_payloads_count())
89+
}
90+
}
91+
92+
/// Partition validators 0..num_validators into `groups` disjoint bitfields,
93+
/// assigning validator `i` to group `i % groups`. Every group is non-empty
94+
/// (groups is capped at the validator count) and the union covers every
95+
/// validator exactly once.
96+
fn participant_groups(num_validators: u64, groups: u64) -> Vec<AggregationBits> {
97+
let groups = groups.clamp(1, num_validators);
98+
(0..groups)
99+
.map(|group| {
100+
let mut bits = AggregationBits::with_length(num_validators as usize)
101+
.expect("validator count is within the bitlist limit");
102+
for index in (group..num_validators).step_by(groups as usize) {
103+
bits.set(index as usize, true)
104+
.expect("index is within the bitlist length");
105+
}
106+
bits
107+
})
108+
.collect()
109+
}
110+
111+
/// splitmix64: tiny deterministic generator for placeholder pubkey bytes,
112+
/// avoiding a rand dependency.
113+
fn splitmix64(state: &mut u64) -> u64 {
114+
*state = state.wrapping_add(0x9e37_79b9_7f4a_7c15);
115+
let mut z = *state;
116+
z = (z ^ (z >> 30)).wrapping_mul(0xbf58_476d_1ce4_e5b9);
117+
z = (z ^ (z >> 27)).wrapping_mul(0x94d0_49bb_1331_11eb);
118+
z ^ (z >> 31)
119+
}
120+
121+
fn synthetic_pubkey(rng_state: &mut u64) -> ValidatorPubkeyBytes {
122+
let mut bytes = [0u8; 52];
123+
for chunk in bytes.chunks_mut(8) {
124+
let word = splitmix64(rng_state).to_le_bytes();
125+
chunk.copy_from_slice(&word[..chunk.len()]);
126+
}
127+
bytes
128+
}
129+
130+
#[cfg(test)]
131+
mod tests {
132+
use super::*;
133+
use ethlambda_types::attestation::validator_indices;
134+
135+
#[test]
136+
fn participant_groups_partition_all_validators() {
137+
for (validators, groups) in [(8u64, 2u64), (8, 3), (5, 8), (1, 1), (4096, 4)] {
138+
let partition = participant_groups(validators, groups);
139+
assert_eq!(partition.len() as u64, groups.min(validators));
140+
let mut seen = vec![0u32; validators as usize];
141+
for bits in &partition {
142+
let indices: Vec<u64> = validator_indices(bits).collect();
143+
assert!(!indices.is_empty(), "every group must be non-empty");
144+
for index in indices {
145+
seen[index as usize] += 1;
146+
}
147+
}
148+
assert!(
149+
seen.iter().all(|&count| count == 1),
150+
"every validator must appear in exactly one group: {seen:?}"
151+
);
152+
}
153+
}
154+
155+
#[test]
156+
fn synthetic_pubkeys_are_deterministic() {
157+
let mut a = 42u64;
158+
let mut b = 42u64;
159+
assert_eq!(synthetic_pubkey(&mut a), synthetic_pubkey(&mut b));
160+
let mut c = 43u64;
161+
assert_ne!(synthetic_pubkey(&mut a), synthetic_pubkey(&mut c));
162+
}
163+
}

0 commit comments

Comments
 (0)