feat(cli): add offline block-building benchmark sub-command

[pablodeymo]

🗒️ Description / Motivation

Adds ethlambda benchmark synthetic — an offline harness that measures block building
exactly as executed when the node proposes, against a reproducible synthetic
workload, with no devnet required.

Second of three (design doc → this → comparable reports). What lands here is the
smallest thing that runs: the real proposer path, driven deterministically, reporting one
row per measured iteration. Aggregate statistics, build provenance and machine-readable
output follow in the next PR, so this one can be reviewed for what it measures rather
than how it formats.

Stacked on #591, which adds the node/benchmark token dispatch this uses. Review
that first; the base moves to main once it lands.

$ make bench
Block-building benchmark — synthetic workload (mock crypto)
  validators=8 warmup_slots=8 iterations=10 proofs_per_data=1 seed=42
  enable_proposer_aggregation=false max_attestations_per_block=3
  ethlambda/v0.1.0/aarch64-apple-darwin/rustc-v1.97.1 os=macos arch=aarch64 threads=14

  iter           compact  select_payloads     stf_simulate   overhead       wall         root
  1              0.000ms          0.002ms          0.015ms    0.068ms    0.085ms   0x7282cc99
  2              0.000ms          0.002ms          0.015ms    0.066ms    0.083ms   0xb9065af0
  3              0.000ms          0.002ms          0.015ms    0.064ms    0.081ms   0x303f6b0f

How to read this

In this order — each piece is understandable without the next:

1. corpus.rs — the workload. Deterministic validators, a genesis store over
   InMemoryBackend, and seed_pool, which fills the pending pool for one slot and
   reports how many entries the next build will see.
2. build_one_slot (mod.rs) — one slot end to end: seed, time the build, import the
   block. Warmup and measured slots run this same path; only whether the sample is kept
   differs, so there is no "am I warming up?" branching inside.
3. PhaseTimer (mod.rs) — start() before the build, finish() after. Two
   readings of the existing phase histogram; the difference between their sample sums is
   the build's phase time, so nothing is added to the hot path.
4. run_synthetic (mod.rs) — validate, set up, loop over slots, report. 46 lines.
5. report.rs — the types and the per-iteration table.

What Changed

File	Change
bin/ethlambda/src/benchmark/mod.rs	Harness driver: clap options + validation, run_synthetic's slot loop, build_one_slot for one slot's work, and PhaseTimer for per-phase attribution
bin/ethlambda/src/benchmark/corpus.rs	Seeded synthetic corpus: genesis store over InMemoryBackend, deterministic pubkeys via splitmix64, and per-slot pool seeding in fixed insertion order, which also rejects a batch the pool would evict whole
bin/ethlambda/src/benchmark/report.rs	Params/Environment/Sample types and the human-readable per-iteration table
bin/ethlambda/src/command.rs	benchmark joins node as a second clap sub-command, so clap lists it in --help and names it in its own usage lines. The node payload becomes Box<CliOptions> now that a much smaller variant sits beside it
bin/ethlambda/src/main.rs	main becomes synchronous and dispatches; only the node path enters the tokio runtime (run_node carries the #[tokio::main] attributes). Benchmark logs go to stderr at WARN so the report owns stdout
crates/storage/{lib,store}.rs	Export NEW_PAYLOAD_CAP so the harness rejects a --proofs-per-data batch the pending pool would evict whole
Makefile	make bench (override BENCH_ARGS to customize)

Correctness / Behavior Guarantees

• cli.rs is not touched by this PR and the node runtime is unchanged. The harness
  arguments live in their own Args group; the node's stay plain PathBuf/String, so
  clap keeps emitting its own missing-argument errors.
• It measures the production path, not a copy: the harness enters through
  produce_block_with_signatures, the same function BlockChainServer::propose_block
  calls, and seeds the pending pool so the proposal tick promotes it exactly as on a
  live node.
• Determinism: same seed + params → identical per-iteration block roots. Verified
  across repeated runs; the roots are printed so a baseline-vs-optimized diff proves an
  optimization changed only speed, not attestation selection. The harness never reads the
  wall clock into results.
• Exact phase attribution with zero hot-path changes: per-iteration
  select_payloads/compact/stf_simulate come from the sample sums of the existing
  lean_block_proposal_attestation_build_phase_seconds histogram, deltaed between
  iterations, with a per-phase assertion that the count advanced by exactly one.
  overhead is the clamped remainder of wall minus the phases.
• The benchmark never starts the tokio runtime, so it cannot park a worker thread for the
  duration of a CPU-bound run.

Tests Added / Run

• corpus.rs: participant groups partition every validator; synthetic pubkeys are
  deterministic for a seed.
• command.rs: the benchmark token parses with no node argument, rejects node flags, and
  its usage line names the sub-command.
• Verified by hand: make bench; identical block-root sequences across two runs at the
  same seed; ethlambda --genesis config.yaml still failing with clap's own
  missing-argument list.
• make fmt, make lint, make test (576 tests, 30 suites) — all clean.

Related Issues / PRs

• Stacked on feat(cli): accept a node sub-command for running the node #591; design doc in the accompanying docs PR
• Followed by the comparable-reports PR
• Splits the now-closed feat(cli): add offline block-building benchmark subcommand #497 / feat(cli): add offline block-building benchmark sub-command #593
• Related to Block building optimizations #465

✅ Verification Checklist

• Ran make fmt — clean
• Ran make lint (clippy with -D warnings) — clean
• Ran make test (cargo test --workspace --profile release-fast) — all passing

[github-actions]

🤖 Kimi Code Review

This is a well-structured benchmarking PR that correctly isolates the block-building path. The code is clean, properly documented, and follows Rust idioms. Below are specific observations.

Code Correctness & Safety

bin/ethlambda/src/benchmark/corpus.rs:82

store.insert_new_aggregated_payloads_batch(entries);

The batch insert assumes the store's internal ordering is deterministic. Ensure insert_new_aggregated_payloads_batch preserves insertion order (or document if it sorts), since the comment notes that "pool insertion order pins within-entry proof choice."

bin/ethlambda/src/benchmark/mod.rs:114

eyre::ensure!(
    options.proofs_per_data as usize <= NEW_PAYLOAD_CAP,
    "--proofs-per-data {} exceeds the pending-pool capacity..."
);

Good defensive check. However, NEW_PAYLOAD_CAP is 64, and proofs_per_data is u64, but the cast to usize could theoretically truncate on 32-bit platforms. Given the clap constraint range(1..=4096) on num_validators and the logical constraint that proofs_per_data ≤ validators, this is safe in practice, but consider using try_into() for explicitness.

bin/ethlambda/src/benchmark/mod.rs:129

let total_slots = options
    .warmup_slots
    .checked_add(common.iterations)
    .ok_or_else(|| eyre::eyre!("--warmup-slots plus --iterations overflows u64"))?;

Correct overflow handling.

bin/ethlambda/src/benchmark/mod.rs:159

proof: MultiMessageAggregate::default(),

In mock-crypto mode this is fine, but verify that MultiMessageAggregate::default() produces a valid "empty" aggregate that doesn't trigger SSZ serialization panics when the block is later imported via on_block_without_verification.

Consensus & Cryptographic Considerations

bin/ethlambda/src/benchmark/corpus.rs:12

const GENESIS_TIME: u64 = 1_700_000_000;

Hardcoding genesis time is correct for deterministic benchmarking. Ensure this doesn't conflict with any time-based fork logic if the benchmark is later extended to simulate forks across long timespans.

bin/ethlambda/src/benchmark/corpus.rs:48-52

let validators = (0..self.num_validators)
    .map(|index| Validator {
        attestation_pubkey: synthetic_pubkey(&mut rng_state),
        proposal_pubkey: synthetic_pubkey(&mut rng_state),
        index,
    })

The deterministic RNG is appropriate here. Verify that synthetic_pubkey generates bytes that are valid public key encodings for the spec (even if signatures aren't verified), or ensure that mock_crypto mode truly bypasses all deserialization. If the leanVM or XMSS code attempts to parse these 52-byte arrays as pubkeys and fails, the benchmark may panic.

bin/ethlambda/src/benchmark/mod.rs:167

on_block_without_verification(&mut store, signed_block)

Documented why verification is skipped (self-generated blocks). Ensure this function doesn't skip state-transition validation that could corrupt the store for subsequent iterations. The comment says it imports "outside the measured span," which is correct methodology.

Performance & Memory

bin/ethlambda/src/benchmark/mod.rs:251-252

.flat_map(|family| family.get_metric())
.filter_map(|metric| { ... })

This iterates all metrics in the global registry. If the registry grows large, consider caching the histogram lookup by name, though for a benchmark tool this is negligible.

bin/ethlambda/src/benchmark/corpus.rs:103-107

fn splitmix64(state: &mut u64) -> u64 {
    *state = state.wrapping_add(0x9e37_79b9_7f4a_7c15);
    ...
}

Consider marking this #[inline] since it's called in a tight loop during validator generation.

Error Handling & Robustness

bin/ethlambda/src/benchmark/mod.rs:137

eyre::ensure!(
    store.new_aggregated_payloads_count() > 0,
    "seeded attestations were evicted from the pending pool at slot {slot}..."
);

Excellent check. Silent eviction would produce misleading benchmark results (empty blocks).

bin/ethlambda/src/benchmark/mod.rs:223-229

eyre::ensure!(
    observations == 1,
    "phase '{phase}' was observed {observations} times during one build (expected 1)..."
);

The assumption of single-threaded execution is valid now but fragile. Add a comment warning that this check will fail if produce_block_with_signatures is ever parallelized internally.

CLI & UX

bin/ethlambda/src/command.rs:95-96

args.drain(..2);
let argv = std::iter::once(OsString::from("ethlambda benchmark")).chain(args);

Clever handling to get clap to render ethlambda benchmark in usage strings. Ensure this doesn't panic if args has fewer than 2 elements (though get(1) check above guards this).

bin/ethlambda/src/main.rs:41-53
The separation of sync benchmark vs. async node runtime is correct. Starting the tokio runtime only for the node avoids parking threads during CPU-bound benchmarking.

Testing

bin/ethlambda/src/benchmark/corpus.rs:123-141
The participant_groups_partition_all_validators test thoroughly validates the partitioning logic including edge cases (more groups than validators).

bin/ethlambda/src/command.rs:185-247
Good test coverage for the new subcommand parsing, including rejection of node-specific arguments in benchmark mode.

Minor Suggestions

1. bin/ethlambda/src/benchmark/report.rs:144

   fn format_ms(seconds: f64) -> String {
       format!("{:.3}ms", seconds * 1e3)
   }

   Consider handling negative zeros (-0.000ms) which can occur with floating point rounding, though the max(0.0) clamp in mod.rs:178 mostly prevents this.

2. **`bin/eth

---

Automated review by Kimi (Moonshot AI) · kimi-k2.5 · custom prompt

[github-actions]

🤖 Codex Code Review

1. bin/ethlambda/src/benchmark/mod.rs:103 rejects some valid workloads because it compares the raw --proofs-per-data argument to NEW_PAYLOAD_CAP, but the actual number of seeded proofs is clamped to min(proofs_per_data, num_validators) in bin/ethlambda/src/benchmark/corpus.rs:78. For example, --num-validators 8 --proofs-per-data 100 only seeds 8 aggregates, yet the benchmark aborts as if it would overflow the pending pool. The guard should use the effective seeded proof count, not the raw CLI value.

I didn’t find a production consensus/security regression in the new proposer-path benchmark wiring or the node/benchmark command split. The main residual risk is limited verification: I could not run cargo check/tests in this sandbox because Cargo/rustup tried to write under read-only ~/.cargo / ~/.rustup (failed to create directory ... Read-only file system).

---

Automated review by OpenAI Codex · gpt-5.4 · custom prompt

[github-actions]

🤖 Claude Code Review

Review: ethlambda benchmark synthetic (PR #595)

Overall this is well-scoped, non-invasive tooling: it doesn't touch cli.rs or the node runtime's parsing/execution paths, and it drives the real produce_block_with_signatures entry point rather than a re-implementation. I traced the key claims against the current code and they hold up:

• Proposer round-robin matches production. bin/ethlambda/src/benchmark/mod.rs:340 (slot % options.num_validators) is identical to is_proposer/current_proposer in crates/blockchain/state_transition/src/lib.rs:238-246.
• Pool seeding/promotion ordering is correct. seed_pool inserts into new_payloads before produce_block_with_signatures is called; promotion to known_payloads happens internally via get_proposal_head → accept_new_attestations (crates/blockchain/src/store.rs:887-897), so by the next iteration the new-pool is empty again before the next seed_pool call — no cross-iteration accumulation that would silently trip the NEW_PAYLOAD_CAP eviction guard.
• Determinism. synthetic_pubkey's struct-literal field evaluation order (attestation_pubkey, then proposal_pubkey) is guaranteed by Rust to run left-to-right, so the shared &mut rng_state produces a reproducible sequence per seed.
• ProposerConfig is Copy, so reusing it by value across the loop in run_synthetic is fine.

Minor observations (non-blocking)

1. --enable-proposer-aggregation is currently unreachable. run_synthetic (mod.rs:299-302) hard-requires common.mock_crypto, and mock_crypto is conflicts_with = "enable_proposer_aggregation" (mod.rs:281). So today, any invocation passing --enable-proposer-aggregation fails clap's conflict check before ever reaching the mock_crypto check — the flag exists in --help but has no valid combination yet. That's presumably intentional (real-crypto lands in a later milestone per the PR description), but it may be worth a one-line note in the flag's help text (or hide = true until it's usable) so users don't waste time discovering the conflict via trial and error.

2. Global Prometheus registry as the measurement channel (phase_snapshot/phase_deltas, mod.rs:424-469) is a reasonable no-hot-path-change approach for a single-threaded, single-process CLI run, but it's worth flagging as a coupling point: if benchmark ever grows a mode that runs iterations concurrently (e.g., to model network overlap) or reuses the same process for multiple sub-runs, the shared global histogram would need explicit synchronization/reset. Not an issue for the current sequential-loop implementation.

3. NEW_PAYLOAD_CAP widened to pub (crates/storage/src/lib.rs, store.rs:127) turns an internal tuning constant into public API surface consumed by the CLI crate. Fine for now, but future changes to the pool's capacity policy (e.g., making it configurable or dynamic) will need to keep this benchmark-facing constant in sync.

4. Test fragility nit: benchmark_parses_without_any_node_argument (command.rs) asserts on format!("{options:?}").contains("num_validators: 4") — a Debug-string substring match tied to derived field ordering/formatting. Low risk given it's a #[derive(Debug)] struct that's unlikely to change shape casually, but a more structural assertion (matching into the parsed field directly) would be more robust to formatting changes.

5. Overflow/edge handling is properly guarded: warmup_slots.checked_add(iterations) avoids silent wraparound, and the proofs_per_data <= NEW_PAYLOAD_CAP / post-seed non-empty checks catch misconfiguration early with clear error messages rather than silently producing empty-attestation samples.

Security / consensus-layer impact

None — this is offline dev tooling with no network exposure, no new signature-verification code paths (mock-crypto mode explicitly gates out the unimplemented real-crypto path), and it uses on_block_without_verification, the same helper already used by the fork-choice spec test harness, so it doesn't introduce a new way to bypass verification in production code.

No blocking issues found.

---

Automated review by Claude (Anthropic) · sonnet · custom prompt