Chunked resumable snapshot state-sync (Phase 2 of #132)

[t-bhendarkar]

Phase 2 of #132. Phase 1 (streaming install-in-place, bounded memory, crash marker) shipped in #201 and fixed the OOM that kept large joiners from installing. Phase 1 is NOT resumable: any mid-download failure wipes the partial state and re-fetches the whole snapshot. That is correct and cheap at current scale, but a liveness gap in production when:

• the link is flaky / WAN and drops mid-download,
• state is large (multi-GB) and one download can't finish in a single window,
• a serving peer restarts while streaming.

In those cases every attempt redoes the whole transfer and can fail at the same point indefinitely. This issue adds resumability: progress survives a crash and a reboot re-fetches only the missing pieces.

Why it needs chunked, content-addressed snapshots

A snapshot is a consistent checkpoint at round R with an ack-signed state_merkle_root. Naive offset-resume forks the chain: by reboot the serving peer has advanced to R' and re-materialises a different snapshot; splicing a prefix from R onto a suffix from R' matches neither root. Resume requires a stable, reproducible snapshot identity plus immutable, independently-verifiable pieces , the Cosmos state-sync / Geth snap-sync model.

Design (summary)

• Manifest: identifies the snapshot at R , chunk count, per-chunk hash, the aggregate consensus roots, committee/acks (everything Phase 1's header carries + the chunk list). Fetched first, ack-quorum + chain-of-trust verified, then the joiner commits to this chunk set.
• Deterministic chunking: total order over all snapshot rows (fixed phase order, canonical PK order within a phase); chunk K = rows [K*N, (K+1)*N); chunk_hash[K] = shake256(canonical bytes). Byte-identical across every honest peer at R, so any chunk is fetchable from any peer and verified by hash. Two integrity layers kept separate: chunk hashes (wire/resume) and consensus roots (over the assembled whole, ack-signed).
• Serving (the real new subsystem): persist chunked snapshot files at periodic checkpoint heights, retain the last N. Serving a chunk = read from file. Survives peer restart, holds no DB transaction. Fallback: on-demand re-materialisation for idle nets (chunk-hash check rejects a stale chunk).
• Protocol (/tip/state-snapshot/2.0.0, negotiated alongside 1.0.0 for rolling deploy): SnapshotManifestRequest -> SnapshotManifest, SnapshotChunkRequest{round,chunk_index} -> Phase-1 frames for that chunk.
• Joiner: fetch manifest, persist {round, installed_chunks bitset}, wipe once, loop fetching missing chunks (install-in-place via the Phase-1 engine, verify each chunk hash, mark done), peer-switch if a peer drops R. Go-live only when all chunks present + all roots + ack-quorum verify (FORK TRAP preserved).
• Crash recovery: boot recovery resumes the chunk loop, re-requesting only missing chunks; wipe + fresh manifest only if R is no longer retained anywhere.

Reuse vs. new

Reused from Phase 1 (no rewrite): streamFrames, kind-tag framing, _boundedFrameStream, install-in-place batching + backpressure, ack-quorum, _verifyRotationChain, readiness gate, fork-trap discipline, crash marker + boot recovery scaffolding, canonical serve iterators. New: chunking + manifest, manifest/chunk protocol, persisted snapshot-file subsystem, joiner chunk-progress + resume + peer-switch, verification re-wire over the assembled set, resume harness.

Phased tasks (~2-3 focused weeks)

• Deterministic chunking + manifest proto + chunk-hash builder (~1d)
• Manifest/chunk request-response protocol, serve + receive (~1-2d)
• Persisted snapshot-file subsystem: create job, retention/GC, serve-from-file, config, NODE_REQUIREMENTS (~4-6d, largest piece)
• Joiner chunk-progress persistence + resume loop + peer-switch + boot recovery (~2d)
• Verification re-wire over assembled chunks, keep fork trap (~1-2d)
• Harness: resume-after-crash-at-chunk-K, out-of-order, peer-switch mid-download, large-snapshot, manifest-divergence (~2-3d)
• Cluster validation + deploy (~1d)

Open decisions

• Persisted files vs on-demand as the first cut (recommend: persisted files).
• SNAPSHOT_INTERVAL / SNAPSHOT_RETAIN / chunk_size_rows defaults, tied to NODE_REQUIREMENTS disk min-spec.
• Bump protocol id to 2.0.0 (register both during rollout).

Full design note lives in my-notes/streaming-snapshot-resume.md.