fix(erigon): bound RAM in commitment + snapshot-write phases (OOM fix at scale)#109
Open
CPerezz wants to merge 6 commits into
Open
fix(erigon): bound RAM in commitment + snapshot-write phases (OOM fix at scale)#109CPerezz wants to merge 6 commits into
CPerezz wants to merge 6 commits into
Conversation
… at scale)
Generating a large trie (e.g. a 100 GB account-heavy alloc, ~613 M EOAs)
OOM-killed the Erigon snapshot writer. The slot stream already stayed
bounded; the OOM lived in the unlogged post-stream phases, which held
O(total-keys) in RAM. Two distinct causes, the second only visible at scale:
1. Commitment held every branch/key in RAM: an in-memory mergedBranches map
(Θ(total-keys)) plus upstream's Updates.keys dedup map (~30-65 GB at
100 GB) plus an etl working set on /tmp (tmpfs = RAM).
2. The snapshot writer os.ReadFile'd each .kv into anonymous heap; Phase 5b
builds domains concurrently, so the 28 GB accounts + 44 GB commitment .kv
loaded at once (~72 GB) → a second OOM at ~120 GiB anon-rss.
Fixes:
- commitment: replace mergedBranches with a live read-write Pebble branchStore
(branchstore.go); chunked first-serial-then-concurrent Process bounds
Updates.keys per chunk; etl spills to cfg.DBPath (real disk) not tmpfs;
ComputeGenesisRoot streams branches into a write-once .kv.
- seg.Decompressor: mmap the .kv read-only (munmap on Close) instead of
os.ReadFile, so the bytes are reclaimable file-backed pages (anon O(1)).
- Phase-5b index builders: btindex offsets → bufio spill; recsplit bucket
collector → streamsort (seq-suffix preserves collision detection);
WriteDomain defer-Close leak fix.
- add --autofill-profile=accounts (streamed pure EOAs) so an account-dominated
large trie is reachable without materializing a sequential_eoas spec.
- env tunables: STATE_ACTOR_{COMMITMENT_CHUNK_KEYS,ERIGON_WORKERS,
COMMITMENT_CACHE_GB,BRANCH_CACHE_GB}.
Consolidates the 10-commit fix branch. Validated: cgo commitment tests (H4
erigon HPH root == geth MPT root; ChunkedVsSingle root==single) + a 100 GB
account-heavy bench (exit 0, 65.9 GiB peak, root 0x31e0bfb1…).
… dedupe, madvise Root-preserving perf follow-up (no RNG/content change; cross-client root pinned by TestH4 + ChunkedVsSingle). None raises RAM; several lower it. - commitment: cap the SERIAL first chunk at min(firstChunkKeys=128K, commitmentChunkKeys). Chunk 0 runs single-core to seed the root branch; at the full chunk width it was ~12% of the whole fold (serial is ~16x/key). 128K keys already span all 16 first nibbles. Root unchanged (chunk boundaries don't affect the fold) — new ChunkedVsSingle assertion with firstChunkKeys=2 < chunk=5. - snapshot_cgo: Close commitmentInputStore immediately after ComputeGenesisRoot (last use) instead of at function scope — frees its block cache + memtable + ~tens-of-GB spill dir BEFORE the mmap-heavy Phase-5b write. Idempotent vs the deferred Close. - generation: hash account code ONCE in encodeEntity and thread it into the commitment Update via new EncodeAccountUpdateCodeHash — was keccak'd twice (snapshot CodeHash + EncodeAccountUpdate) for every code-bearing entity. - seg.Decompressor: MADV_SEQUENTIAL after mmap — aggressive read-ahead for the front-to-back Pass-2 scan and lets the kernel drop read pages sooner, lowering file-RSS on a 44 GiB .kv. Advisory; byte output unchanged.
CPerezz
force-pushed
the
fix/erigon-oom-upstream
branch
from
e2bea25 to
392d5e1
Compare
The commitment domain is the LARGEST (~44 GB .kv at 100 GB, plus the only recsplit MPHF over ~nBranches keys) yet ran SERIALLY after the accounts/storage/ code fan-out, leaving the machine ~1-core-busy during its build. It depends only on branchesStore + keyStateValue (both ready from the fold), not on the other domains, so run it as a 4th goroutine in the same fan-out — the Writer is immutable and each domain writes its own files. Phase-5b wall time drops toward max(domains) instead of sum. Byte output unchanged.
…old (item 2) The three account/storage/code domain writes depend only on the finalized stores + counts, NOT on the commitment fold's result — yet they ran AFTER the 2-4h fold in Phase 5b, on cores the 16-way fold left idle. Start their fan-out BEFORE ComputeGenesisRoot so ~sequential compression overlaps the fold; only WriteCommitment (needs branches + keyStateValue) stays queued after. The Writer is immutable and each domain writes its own files, so no races; output bytes are unchanged (same inputs, reordered concurrency only).
…(item 6b) seg.Compressor.Compress rescanned the entire .idt (28-44 GiB per domain at 100 GB) solely to build the word-length histogram (posMap) + emptyWordsCount. Both depend only on each word's length, which AddWord already sees — accumulate them incrementally there and drop the Pass-A ForEach. One fewer full pass over the intermediate per domain. Byte output unchanged (the histogram fully determines the Huffman tree); seg goldens green.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
Generating a large trie (e.g. a 100 GB account-heavy alloc, ~613 M EOAs) OOM-kills the Erigon
snapshot writer. The slot stream already stays RAM-bounded; the OOM lives in the unlogged
post-stream phases, which held
O(total-keys)in RAM.mainstill exhibits this (the oldin-memory
mergedBranchesmap + single-argComputeGenesisRoot). Two distinct causes — thesecond only visible at ≳100 GB scale:
mergedBranchesmap (Θ(total-keys))Updates.keysdedup map (~30–65 GB at 100 GB) + an etl working set on/tmp(tmpfs = RAM).
os.ReadFile'd each.kvinto anonymous heap. Phase 5b builds domainsconcurrently, so the ~28 GB accounts
.kv+ ~44 GB commitment.kvloaded at once (~72 GB ofhard heap) → a second OOM (
anon-rss~120 GiB on a 125 GiB host).Fixes
mergedBrancheswith a live read-write PebblebranchStore(
branchstore.go); a chunked first-serial-then-concurrentProcessboundsUpdates.keysperchunk; the etl spills to the datadir (real disk) instead of tmpfs;
ComputeGenesisRootstreamsbranches into a write-once
.kv.seg.Decompressor: open each.kvREAD-ONLY via mmap (munmap onClose) instead ofos.ReadFile, so the bytes are reclaimable, file-backed pages — anon RSS staysO(1)regardlessof
.kvsize.btindexoffsets → bufio spill;recsplitbucket collector →streamsort (a
seqsuffix preserves fingerprint-collision detection);WriteDomaindefer-Closeleak fix.
--autofill-profile=accounts(streamed pure EOAs) so an account-dominated large trie isreachable without materializing a
sequential_eoasspec (a separate pre-Phase-4 OOM).STATE_ACTOR_{COMMITMENT_CHUNK_KEYS,ERIGON_WORKERS,COMMITMENT_CACHE_GB,BRANCH_CACHE_GB}.Validation
TestH4_HexPatriciaHashed_MatchesMPT— erigonHPH root == geth MPT root (
0x639b726b…, cross-client invariance);ChunkedVsSingle— chunkedroot == single-Process root (
0xbfee1b9e…);IncludesRootBranch.feat/erigon-client, the pre-rebase source): exit 0, no OOM,peak RSS 65.9 GiB (was 120 GiB → OOM-killed), root
0x31e0bfb1ca7a93eddc08f8051d78e2bb66977d470eea82fa36eb597f0c58af0f, 613,566,756 EOAs /232,164,579 branches.
I'll post the result as a comment.
Notes
main(was previously stacked onfeat/erigon-client).helpers_test.go'sComputeGenesisRootFromAccountsmoved intocommitment.go;main'sProgressinstrumentation is preserved.root-preserving) is planned as a subsequent PR.