Skip to content

Commit 1be9995

Browse files
authored
feat(aggregation): score recursive aggregation through a per-aggregator subnet window (lambdaclass#613)
## Summary Every aggregator on the network currently does the same aggregation work. In a healthy slot all validators vote for the same head, so there is one hot `AttestationData`; candidates are scored deterministically from head state, and `select_proofs_greedily` picks the two highest-coverage children from a pool every aggregator sees, since aggregates gossip on one global topic. So all aggregators select the same two children and produce the same merged proof, and every copy of that leanVM work past the first is wasted. If every aggregator anchors on the pool's best proof `r0`, then aggregator `i` publishes `out_i = r0 ∪ x_i`, and `out_i ∪ out_j = out_i ∪ x_j`. Merging two published proofs gains nothing over merging one of them with a raw pool proof. This gives each aggregator a **duty subnet** and a **window** of subnets starting there, used as a scoring lens on child selection, so different aggregators merge different children. ## Design - **Window**: the contiguous cyclic run of subnets `{s, s+1, ...}` an aggregator is responsible for, starting at its duty subnet. - **Scoring lens, not a filter**: a child is valued by the validators it newly covers whose subnet is inside the window. A proof straddling the boundary stays usable for its in-window part; one lying wholly outside scores zero. A selected child still contributes *all* of its participants to `covered`, which keeps the marginal-coverage score honest across greedy rounds. - **Width from the anchor**: `min(2 * reach(anchor), C)`, where the *anchor* is the largest-coverage proof in the candidate's pool that touches the aggregator's own duty subnet, and a proof's *reach* is how many distinct subnets it touches. Coverage rather than reach picks the anchor, so a sparse proof holding one validator in each of many subnets no longer sets the width for everybody. Requiring the anchor to touch the duty subnet makes "no anchor" mean "no peer has covered my subnet", which is exactly when this node's raw signatures are irreplaceable: the window then sits at its narrowest and the aggregator works on those instead of merging proofs it cannot add to. Deriving the width rather than choosing it is essential either way, since windows nest and "widest viable window" would collapse to the full committee set for everyone. - **Duty subnet**: the first `--aggregate-subnet-ids` value, else the lowest subscribed subnet, else 0. Logged at startup so a collision is diagnosable. The node refuses to start on an `--aggregate-subnet-ids` value at or above the committee count: subscriptions use the raw id while the window reduces it, so an out-of-range value would subscribe to a topic no validator publishes on and aggregate as a subnet the node does not listen to. - **`--skip-redundant-aggregation`** (opt-in): an aggregator sits out any candidate whose derived width it does not own this slot (`duty_subnet % w == slot % w`), and the freed job goes to the next-best `AttestationData` rather than to a narrower merge of the same one. Ownership rotates with the slot, so no duty subnet is permanently the one sitting out, and width 1 is owned by everyone, so a candidate with no anchor on this node's subnet is never skipped. - **Deployment precondition**: every subnet below the committee count needs an aggregator holding it as its duty subnet. Ownership rotates, but a width can have *no* owner in a slot on a sparser placement even with every node healthy: at duty subnets `{0, 2}` and committee count 4, nothing owns width 4 in an odd slot, and the fallback is off, so that merge level is dropped for the slot. Leave the flag unset on a placement that does not cover every subnet. ## Safety - **No consensus impact.** Nothing in attestation processing, block building, or fork choice is touched. A windowed aggregate binds exactly `raw_ids ∪ accepted_child_ids` and its bits derive from that same set, so it is valid with fewer participants: less fork-choice weight, never wrong weight. - **No coverage regression.** A window is contiguous but the pool need not be contiguous in subnet space, so a strided aggregator placement could leave a window holding one proof and drop a merge the unwindowed selection would have made. When a windowed selection is not viable it falls back to the full committee set, so the window can only improve on the old selection, never regress below it. The fallback is disabled under `--skip-redundant-aggregation`: every width below the committee count has several owners, so retrying there would rebuild exactly the duplication the flag buys away. - **`attestation_committee_count = 1` is a provable no-op**: the window contains every validator, so in-window coverage equals total coverage and the break condition is identical to before, tie-break order included. - **Mixed-network safe.** No wire-format, topic, or fork-digest change. A `main` node's wide proof can raise a branch node's anchor reach where it touches that node's duty subnet, which opens its window, so a partial rollout makes the feature weaker rather than inconsistent. ## Known trade-off Because the anchor must touch the duty subnet, the derived width is no longer uniform across the network. Two aggregators reading one lopsided pool can derive different widths, so their windows nest rather than tile. That costs a round of climbing, not correctness, and it is the direct price of tying the window to work the aggregator can actually contribute to. ## Cadence, in practice There is one aggregation session per slot, and the current slot's pool is empty at snapshot time since produced aggregates are held until the interval-2 boundary. So the window bites on the stale candidate, and a data root gets about one windowed merge rather than a multi-round climb. The widening matters across slots for a data root that stays live. That emptiness is a timing expectation, not an invariant. A peer's aggregate for the current slot landing before this node's snapshot, under clock skew or a session started early via `EarlyAggregationCheck`, gives the current-slot candidate an anchor and a width of 2; under `--skip-redundant-aggregation` the duty subnets that do not own width 2 then sit that slot out and their raw signatures miss the next block. On the intended topology (distinct duty subnets, distinct subscriptions) a round-one peer proof never touches this node's subnet, so it holds anyway. Overlapping subscriptions are where it fails. ## Metrics - `lean_aggregation_window_width` (Histogram): width derived per candidate. Climbs from 1 as the anchor climbs; pinned at the committee count means the window no longer restricts selection. Stuck at 1 while the network is aggregating means the pool holds nothing on this node's duty subnet, so it is only aggregating its own raw signatures. - `lean_aggregation_skipped_redundant_total`: candidates handed to another duty subnet by the redundancy-skipping rotation. Only increments with the flag on. - `lean_aggregation_window_fallback_total`: merges the window would have dropped, recovered by retrying selection at the full committee set. Counts recoveries, not attempts: a candidate no window could have made viable, a lone raw signature or a single-proof group, never reaches the retry. A persistently rising value therefore means the aggregator placement is too sparse for the committee count. Stays flat entirely under `--skip-redundant-aggregation`. ## Test Plan - [x] `cargo test --workspace --profile release-fast`: 695 tests pass - [x] `make lint` clean, `make fmt` clean, `make docs` builds - [x] Four-aggregator reduction pinned end to end: round 1 produces four distinct proofs, round 2 reaches the full validator set - [x] Mid-climb widening covered at committee count 8, where the window grows but stays a proper subset - [x] The strided-placement regression has a test that fails without the fallback - [x] Single-committee no-op pinned - [x] Anchor selection pinned: a duty subnet the pool does not reach gets width 1, a sparse wide proof loses to a denser narrow one, a coverage tie falls to reach so pool order does not matter - [x] A skipped candidate hands its job budget to the next-best `AttestationData` - [x] The full-width retry is skipped where it is provably a no-op: a full-width window, an empty proof pool, a structureless committee - [x] An `--aggregate-subnet-ids` value at or above the committee count is refused at startup, past the first id too - [ ] Devnet: run all-ethlambda with `attestation_committee_count = 4` and aggregators on **distinct** duty subnets, confirm `lean_aggregation_window_width` climbs rather than pinning at 4, fleet-wide aggregation CPU drops against a control, and finality is unaffected - [ ] Devnet: confirm `lean_aggregation_window_fallback_total` stays flat on the intended placement ## Notes for review - The default proposer path (`keep_best_proof_per_data`) keeps one proof per `AttestationData` and drops the rest. Aggregators now emit proofs with distinct coverage rather than near-identical ones, so that path may drop more useful coverage than before. Bounded, since the window is sized to fit two children of the anchor's current reach, so a windowed proof is the same size as before; `--enable-proposer-aggregation` removes the exposure entirely. Declared out of scope here but worth measuring. - With `--skip-redundant-aggregation` set and no explicit `--aggregate-subnet-ids`, every aggregator on a subnet-spanning topology derives duty subnet 0 and sits out in lockstep instead of taking turns. The node warns at startup when that combination is configured. - The raw-signature guarantee is structural rather than a hard floor: "no anchor" means no peer covered our subnet. It is not airtight when two aggregators share a subnet and hold different slices of it, which no supported topology does today.
1 parent 0a90340 commit 1be9995

7 files changed

Lines changed: 1816 additions & 46 deletions

File tree

bin/ethlambda/src/cli.rs

Lines changed: 37 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -83,8 +83,45 @@ pub(crate) struct NodeOptions {
8383
pub(crate) attestation_committee_count: Option<u64>,
8484
/// Subnet IDs this aggregator should subscribe to (comma-separated).
8585
/// Requires --is-aggregator. Defaults to the subnets of the node's validators.
86+
/// Every ID must be below --attestation-committee-count; the node refuses to
87+
/// start otherwise, since a higher ID names a topic no validator publishes on.
88+
///
89+
/// The first ID is also this node's aggregation duty subnet: where its
90+
/// aggregation window starts, and what --skip-redundant-aggregation
91+
/// rotates ownership over. Order matters, so give co-located aggregators
92+
/// different first IDs. Unset, the duty subnet falls back to the lowest
93+
/// subscribed subnet, which is the same value on every node whose
94+
/// validators span all subnets.
8695
#[arg(long, value_delimiter = ',', requires = "is_aggregator")]
8796
pub(crate) aggregate_subnet_ids: Option<Vec<u64>>,
97+
/// Sit out aggregation candidates whose level another duty subnet owns
98+
/// this slot. Requires --is-aggregator.
99+
///
100+
/// By default every aggregator merges proofs for a window of subnets
101+
/// starting at its duty subnet, and windows belonging to neighbouring duty
102+
/// subnets overlap, so some prover work is duplicated. With this flag an
103+
/// aggregator skips a candidate whose width it does not own in the current
104+
/// slot and spends that job on the next-best attestation data instead. The
105+
/// owner rotates with the slot, so no node is permanently the one sitting
106+
/// out, and the narrowest width is owned by everyone, so a candidate whose
107+
/// pool holds nothing on this node's subnet, which is the raw-signature
108+
/// case, is never skipped.
109+
///
110+
/// Worth enabling when leanVM prover CPU is the bottleneck on co-located
111+
/// aggregators.
112+
///
113+
/// Deployment precondition: give every subnet below
114+
/// --attestation-committee-count an aggregator holding it as its duty
115+
/// subnet. Ownership is `duty_subnet % width == slot % width`, so on a
116+
/// sparser placement a width can have no owner at all while every
117+
/// configured node is healthy. With duty subnets {0, 2} at committee count
118+
/// 4, nothing owns width 4 in an odd slot, and since this flag also
119+
/// disables the full-width fallback, that merge level is simply dropped
120+
/// for the slot. The narrower levels still run and the next slot rotates to
121+
/// a different owner, but the loss is structural, not just the cost of a
122+
/// node that is down or late.
123+
#[arg(long, default_value = "false", requires = "is_aggregator")]
124+
pub(crate) skip_redundant_aggregation: bool,
88125
/// Directory for RocksDB storage
89126
#[arg(long, default_value = "./data")]
90127
pub(crate) data_dir: PathBuf,

bin/ethlambda/src/main.rs

Lines changed: 148 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -25,7 +25,7 @@ static ALLOC: tikv_jemallocator::Jemalloc = tikv_jemallocator::Jemalloc;
2525
static malloc_conf: &[u8] = b"prof:true,prof_active:true,lg_prof_sample:19\0";
2626

2727
use std::{
28-
collections::{BTreeMap, HashMap},
28+
collections::{BTreeMap, HashMap, HashSet},
2929
net::{IpAddr, SocketAddr},
3030
path::{Path, PathBuf},
3131
sync::Arc,
@@ -217,6 +217,12 @@ async fn run_node(options: NodeOptions) -> eyre::Result<()> {
217217
attestation_committee_count,
218218
"Loaded attestation committee count"
219219
);
220+
// Checked here rather than in clap: the committee count is only known once
221+
// the CLI flag and the validator config have both been consulted.
222+
validate_aggregate_subnet_ids(
223+
options.aggregate_subnet_ids.as_deref(),
224+
attestation_committee_count,
225+
)?;
220226
ethlambda_blockchain::metrics::set_attestation_committee_count(attestation_committee_count);
221227

222228
let bootnodes = read_bootnodes(&bootnodes_path)?;
@@ -279,12 +285,34 @@ async fn run_node(options: NodeOptions) -> eyre::Result<()> {
279285
// receiver-count guard in `emit` makes every emission a no-op.
280286
let events = EventBus::default();
281287

288+
let aggregation_duty_subnet = resolve_aggregation_duty_subnet(
289+
options.aggregate_subnet_ids.as_deref(),
290+
&subscribed_subnets,
291+
);
292+
info!(
293+
aggregation_duty_subnet,
294+
assigned = options.aggregate_subnet_ids.is_some(),
295+
"Resolved aggregation duty subnet"
296+
);
297+
if options.skip_redundant_aggregation && options.aggregate_subnet_ids.is_none() {
298+
warn!(
299+
aggregation_duty_subnet,
300+
"--skip-redundant-aggregation is set but the duty subnet was derived, not assigned: \
301+
every co-located aggregator whose validators span all subnets derives the same duty \
302+
subnet, so they will sit out in lockstep in the same slot instead of taking turns, \
303+
and the widest level gets no producer at all in most slots. Give each aggregator a \
304+
distinct first --aggregate-subnet-ids value to fix this."
305+
);
306+
}
307+
282308
let blockchain_config = BlockChainConfig {
283309
aggregator: aggregator.clone(),
284310
sync_status_controller: sync_status.clone(),
285311
attestation_committee_count,
286312
gate_duties: !options.disable_duty_sync_gate,
287313
subscribed_subnets: subscribed_subnets.clone(),
314+
aggregation_duty_subnet,
315+
skip_redundant_aggregation: options.skip_redundant_aggregation,
288316
proposer_config: ProposerConfig {
289317
enable_proposer_aggregation: options.enable_proposer_aggregation,
290318
max_attestations_per_block: options.max_attestations_per_block,
@@ -818,13 +846,132 @@ async fn fetch_initial_state(
818846
Ok(store)
819847
}
820848

849+
/// Reject an `--aggregate-subnet-ids` value that names no subnet.
850+
///
851+
/// The flag feeds two consumers that read an out-of-range value differently:
852+
/// the P2P swarm subscribes to the raw id (`attestation_subscription_subnets`
853+
/// passes it through), while the aggregation window reduces it modulo the
854+
/// committee count. `--attestation-committee-count 4 --aggregate-subnet-ids 5`
855+
/// therefore subscribes to a topic no validator publishes on and aggregates as
856+
/// duty subnet 1, which the node does not listen to, with the startup log
857+
/// showing 5 either way. Refusing to start is the only reading of that
858+
/// configuration that cannot silently mean something else.
859+
fn validate_aggregate_subnet_ids(
860+
assigned_subnet_ids: Option<&[u64]>,
861+
attestation_committee_count: u64,
862+
) -> eyre::Result<()> {
863+
let out_of_range = assigned_subnet_ids
864+
.unwrap_or_default()
865+
.iter()
866+
.find(|&&id| id >= attestation_committee_count);
867+
match out_of_range {
868+
None => Ok(()),
869+
Some(id) => Err(eyre::eyre!(
870+
"--aggregate-subnet-ids value {id} is not a subnet: ids must be below \
871+
attestation_committee_count ({attestation_committee_count})"
872+
)),
873+
}
874+
}
875+
876+
/// The subnet this node is responsible for when scoring recursive aggregation.
877+
///
878+
/// Operators assign it explicitly via --aggregate-subnet-ids so co-located
879+
/// aggregators land on different subnets and merge different proofs. Without
880+
/// an assignment, fall back to the lowest subnet this node already listens
881+
/// on: `min` rather than an arbitrary pick because `HashSet` iteration order
882+
/// is not stable and the duty subnet must be.
883+
fn resolve_aggregation_duty_subnet(
884+
assigned_subnet_ids: Option<&[u64]>,
885+
subscribed_subnets: &HashSet<u64>,
886+
) -> u64 {
887+
assigned_subnet_ids
888+
.and_then(|ids| ids.first().copied())
889+
.or_else(|| subscribed_subnets.iter().copied().min())
890+
.unwrap_or(0)
891+
}
892+
821893
#[cfg(test)]
822894
mod tests {
823895
use super::*;
824896
use ethlambda_storage::backend::InMemoryBackend;
825897
use ethlambda_types::constants::DEFAULT_MILLISECONDS_PER_SLOT;
826898
use ethlambda_types::genesis::GenesisValidatorEntry;
827899

900+
/// The duty subnet is the first explicitly assigned subnet, so an operator
901+
/// can place co-located aggregators on different subnets deliberately.
902+
#[test]
903+
fn duty_subnet_prefers_the_first_assigned_id() {
904+
let subscribed = HashSet::from([0u64, 1, 2, 3]);
905+
assert_eq!(
906+
resolve_aggregation_duty_subnet(Some(&[3, 1]), &subscribed),
907+
3,
908+
"the first assigned id wins, not the lowest"
909+
);
910+
}
911+
912+
/// With no assignment, the lowest subscribed subnet is used, which is
913+
/// stable across restarts unlike an arbitrary pick from the set.
914+
#[test]
915+
fn duty_subnet_falls_back_to_the_lowest_subscribed() {
916+
let subscribed = HashSet::from([5u64, 2]);
917+
assert_eq!(resolve_aggregation_duty_subnet(None, &subscribed), 2);
918+
}
919+
920+
/// A node with nothing assigned and nothing subscribed still needs an
921+
/// answer; subnet 0 always exists.
922+
#[test]
923+
fn duty_subnet_defaults_to_zero_with_nothing_to_go_on() {
924+
assert_eq!(resolve_aggregation_duty_subnet(None, &HashSet::new()), 0);
925+
}
926+
927+
/// An empty list is no assignment at all, so the subscription fallback
928+
/// still applies rather than the last-resort zero.
929+
#[test]
930+
fn duty_subnet_treats_an_empty_assignment_as_no_assignment() {
931+
assert_eq!(
932+
resolve_aggregation_duty_subnet(Some(&[]), &HashSet::from([4u64])),
933+
4
934+
);
935+
}
936+
937+
/// An id at or above the committee count names a topic no validator
938+
/// publishes on, and would be reduced to a different subnet by the
939+
/// aggregation window, so the node refuses it rather than running with
940+
/// its subscriptions and its duty subnet disagreeing.
941+
#[test]
942+
fn an_out_of_range_aggregate_subnet_id_is_rejected() {
943+
let err = validate_aggregate_subnet_ids(Some(&[5]), 4)
944+
.expect_err("subnet 5 does not exist at committee count 4");
945+
let message = err.to_string();
946+
assert!(message.contains('5'), "names the offending id: {message}");
947+
assert!(message.contains('4'), "names the bound: {message}");
948+
}
949+
950+
/// The check covers every id, not just the first: only the first becomes
951+
/// the duty subnet, but all of them become gossip subscriptions.
952+
#[test]
953+
fn an_out_of_range_aggregate_subnet_id_is_rejected_past_the_first() {
954+
assert!(validate_aggregate_subnet_ids(Some(&[0, 9]), 4).is_err());
955+
}
956+
957+
/// The bound is exclusive: subnets run 0..committee_count.
958+
#[test]
959+
fn in_range_aggregate_subnet_ids_are_accepted() {
960+
assert!(validate_aggregate_subnet_ids(Some(&[0, 3]), 4).is_ok());
961+
assert!(
962+
validate_aggregate_subnet_ids(Some(&[0]), 1).is_ok(),
963+
"subnet 0 is the only subnet at a committee count of 1"
964+
);
965+
}
966+
967+
/// Nothing assigned is nothing to validate; the duty subnet is then
968+
/// derived from subscriptions, which are already in range by construction.
969+
#[test]
970+
fn an_unset_or_empty_assignment_passes_validation() {
971+
assert!(validate_aggregate_subnet_ids(None, 4).is_ok());
972+
assert!(validate_aggregate_subnet_ids(Some(&[]), 4).is_ok());
973+
}
974+
828975
/// Validator-config snippet matching `lean-quickstart`'s ansible-devnet
829976
/// where networks share a non-default committee count.
830977
const VC_WITH_COMMITTEE_COUNT: &str = r#"

0 commit comments

Comments
 (0)