msf-1: mainnet shadowfork from the jochemnet benchmark snapshots

[qu0b]

Changes over the sepsf-2 copy (e14d8ea). Every EL restores benchmarkoor's
jochemnet/24402727 datadir instead of a canonical snapshot. jochemnet started
from the mainnet 24350000 snapshot and ran 1 s blocks of bloat state up to
24402727 (head ts 2026-01-31). Its head is not on mainnet.

Snapshots (snapshots.ethpandaops.io/jochemnet//24402727/), layouts
checked from the first 64 MB of each tarball:

• geth/besu/nethermind snapshot.tar.zst, reth snapshot-v2.tar.zst (v2
  storage), erigon snapshot-pruned.tar.zst. Sizes 0.7-1.26 TB compressed, so
  the 1.8 TB droplets (DO's largest local disk) stay.
• ethrex snapshot.tar.zst (0.3 TB) is new compared to sepsf. It extracts into
  /data/ethrex/chain-1, which is where ethrex 29 keeps a chain-id-1
  custom-genesis db. It runs --syncmode=full; the May db is migrated on start.
• nimbus-el still has no snapshot and snap-syncs from our peers.
• shadowfork.yaml: snapshot_fetcher_filename moves from play vars (which
  beat group_vars) to a template default, so it can be set per EL.

Genesis (egg 6.2.0, CHAIN_ID=1, mainnet deposit contract):

• jochemnet publishes no _snapshot_eth_getBlockByNumber.json. The one inside
  the erigon tarball is the stale mainnet 24350000 block. SHADOW_FORK_FILE is
  therefore files/shadowfork_block.json, the full-tx eth_getBlockByNumber of
  a restored EL; TODO before playbook.yaml. Tested with a stand-in head: the
  real genesis role keeps osaka/bpo1/bpo2 at mainnet times and puts
  amsterdamTime at genesis + 40 epochs. genesis.ssz builds; a hash-only block
  makes eth2-testnet-genesis fail.
• Mainnet has no EIP-8282 builder contracts, so the devnet-7/8
  deploy-eip8282-contracts assertoor startup test is back. It sends the same
  0x4e59 factory calldata as the sepolia deployment txs; checked with cast
  create2.

Clients:

• nethermind: --config=mainnet, nethermind_db/mainnet, FlatDb (the
  tarball's mainnet/flat layout).
• erigon: keeps --externalcl and --keep.stored.chain.config; db.size.limit
  is 2TB. The sepsf-2 --snap.stop host_vars are dropped: segments are frozen
  up to 24401000, so ~1.7k blocks are left to retire (sepsf-2 had 220k).
• Static EL enodes reset (they named sepsf-2's hosts), plus a note to
  firewall EL p2p fleet-wide: mainnet peers have none of our blocks.

Other:

• Images: nimbus v26.10.0, prysm v7.2.1, ethrex 29.0.1 (released).
• terraform: VPC 10.32.0.0/16.
• Secrets: the metrics/logs usernames and the xatu ingest address are now
  glamsterdam-msf-1.

Launch TODOs (marked TODO(msf-1)): genesis time, gloas_fork_epoch,
files/shadowfork_block.json, shadowfork_el_static_enodes.

🤖 Generated with Claude Code

https://claude.ai/code/session_01NnamtoTL7VJzozbP63tWdo

[redpandabot]

Summary

msf-1 shadows mainnet from the jochemnet benchmark snapshots: per-EL snapshot filenames, chain-id-1 shadowfork genesis with a committed block file, mainnet-specific EL peer isolation, a pre-genesis gate, and a regenerated 41-node inventory. The last commit aligns inventory.ini with the restored 36-validator nodes.tf layout and I verified all node names, validator ranges and terraform-computed regions match. The one open item is the still-absent shadowfork_block.json the genesis role requires.

Issues

• 🟡 ansible/inventories/msf-1/group_vars/all/all.yaml:107 — shadowfork_block.json lookup targets a file not in the repo — see the thread on that line

---

_{Reviewed @ 6cbfefd5
"Weeks of coding can save you hours of planning."}

[redpandabot]

🟡 shadowfork_block.json lookup targets a file not in the repo

This resolves to ansible/inventories/msf-1/files/shadowfork_block.json, but no files/ directory is tracked. The ethereum_genesis role consumes the dict via ethereum_genesis_generator_config_files | dict2items (roles/ethereum_genesis/tasks/generate_genesis.yaml), so the missing file lookup raises and aborts the genesis play before any host runs. It is marked TODO, but until the file is committed playbook.yaml cannot get past genesis.

[redpandabot]

🟡 pre-genesis gate always fails on the nimbusel node

The gate queries ethereum_node:bootnode, which includes lighthouse-nimbusel-1. nimbusel has no snapshot, gets no static peers (shadowfork_el_static_enodes is empty) and its snap-sync needs a post-Amsterdam pivot (nimbusel.yaml), so it cannot be at shadowfork_height before genesis; the loop reports "no RPC answer" (line 59) or a head mismatch and exits 1. Exclude nimbusel from the check (or special-case it) or the gate can never pass as documented.