feat(iota-core): backfill epochs_v2 and seed it on snapshot restore by bingyanglin · Pull Request #11697 · iotaledger/iota

bingyanglin · 2026-06-01T07:40:17Z

Description of change

Makes a gRPC fullnode's epochs_v2 index complete since genesis on every bootstrap path and enforces it at startup: the node closes any gap before services start, or refuses to run instead of silently serving an incomplete index — the consumer side of #11453's EPOCH_INFO file. No object/previous_transaction_checkpoint backfill: the snapshot-publisher node resyncs from genesis instead, enforced by the writer's existing refusal of None rows.

Synchronous startup backfill (iota-node, iota-config). A node that detects a gap (epochs_v2_gap: index short of the last executed closed epoch) fetches only MANIFEST + EPOCH_INFO from the new state_snapshot_read_config, seeds [0, snapshot_epoch], closes any residual above it from local history, seeds the open epoch's row, and re-checks; a remaining gap — or no configured source — aborts startup. Runs before live indexing exists, so the watermark has one writer at a time and the previous design's background task, retry/backoff, and epoch_watermark_lock are gone.
Local indexing fallback (iota-core). When the latest published snapshot lags local execution (delayed snapshot pipeline), index_missing_epochs_locally replays only the missing epochs' closing checkpoints, located via the never-pruned epoch_last_checkpoint_map; best-effort up to the pruning horizon.
Atomic EpochIndexed advance (iota-core). The live path advances the watermark in the same batch as the close-of-epoch row (gap-aware try_advance_epoch_indexed_watermark); reconcile_epoch_indexed_watermark remains only to jump across a seeded prefix.
Restore builds the whole gRPC index store (iota-tool, iota-snapshot, iota-core). download_formal_snapshot tees the restored object stream into the live-state indexers (RestoreWithGrpcIndexes), seeds the epoch rows, and finalizes the store (Watermark::Indexed, then meta — a crash before meta leaves a store the next open wipes and re-inits). The node opens it in place instead of re-indexing the whole restored state; opt out with --skip-grpc-indexes. init and the restore share one indexing implementation (GrpcLiveObjectRestorer; ParMakeLiveObjectIndexer is lifetime-generic now).
Chain-identity gate (iota-snapshot). verify_and_restore_epoch_info rejects a snapshot whose manifest chain_id differs from this node's chain before writing any row.
RestoreEpochInfo trait (iota-snapshot). Separate single-method trait instead of a new Restore method: the two cover different snapshot payloads with different targets, so each call site requires exactly the capability it uses; the unified indexer ([indexer]: startup from formal snapshots #11023) can implement both.
No epochs migration (iota-core). The deprecated epochs CF is dropped without migration: its rows lack the end-of-epoch fields, so they could never satisfy EpochIndexed and the backfill would overwrite them anyway.
Open-epoch seeding (iota-core). initialize_current_epoch_info keys off the open epoch (open_epoch_of): a restore lands on a closing checkpoint, and seeding that checkpoint's own (closed) epoch would leave the open epoch's row permanently missing and wedge the watermark. Its start checkpoint derives from epoch_last_checkpoint_map (new seq-only CheckpointStore accessor; the backwards scan over prunable summaries is deleted).
Default snapshot source (setups). Fullnode setups for mainnet/testnet/devnet ship a default state-snapshot-read-config pointing at the IOTA Foundation buckets; ignored unless the gRPC API is enabled and the index is incomplete.
Misc. Uploader honors state_snapshot_write_config.concurrency (default still 20); get_latest_available_epoch delegates to iota_snapshot::reader::latest_available_epoch; snapshots.mdx documents the restore flag and the backfill setup.

Links to any relevant issues

follow-up (PR-2) of #11453 · part of #11023

How the change has been tested

Basic tests (linting, compilation, formatting, unit/integration tests)
Patch-specific tests (correctness, functionality coverage)
I have added tests that prove my fix is effective or that my feature works
I have checked that new and existing unit tests pass locally with my changes

…pipeline

github-actions · 2026-06-11T10:30:12Z

✅ Vercel Preview Deployment is ready!

View Preview

github-actions · 2026-06-11T12:51:01Z

✅ Vercel Preview Deployment is ready!

View Preview

…11697) # Description of change Makes a gRPC fullnode's `epochs_v2` index complete since genesis on every bootstrap path and enforces it at startup: the node closes any gap before services start, or refuses to run instead of silently serving an incomplete index — the consumer side of #11453's `EPOCH_INFO` file. No object/`previous_transaction_checkpoint` backfill: the snapshot-publisher node resyncs from genesis instead, enforced by the writer's existing refusal of `None` rows. - **Synchronous startup backfill (`iota-node`, `iota-config`).** A node that detects a gap (`epochs_v2_gap`: index short of the last executed closed epoch) fetches only MANIFEST + `EPOCH_INFO` from the new `state_snapshot_read_config`, seeds `[0, snapshot_epoch]`, closes any residual above it from local history, seeds the open epoch's row, and re-checks; a remaining gap — or no configured source — aborts startup. Runs before live indexing exists, so the watermark has one writer at a time and the previous design's background task, retry/backoff, and `epoch_watermark_lock` are gone. - **Local indexing fallback (`iota-core`).** When the latest published snapshot lags local execution (delayed snapshot pipeline), `index_missing_epochs_locally` replays only the missing epochs' closing checkpoints, located via the never-pruned `epoch_last_checkpoint_map`; best-effort up to the pruning horizon. - **Atomic `EpochIndexed` advance (`iota-core`).** The live path advances the watermark in the same batch as the close-of-epoch row (gap-aware `try_advance_epoch_indexed_watermark`); `reconcile_epoch_indexed_watermark` remains only to jump across a seeded prefix. - **Restore builds the whole gRPC index store (`iota-tool`, `iota-snapshot`, `iota-core`).** `download_formal_snapshot` tees the restored object stream into the live-state indexers (`RestoreWithGrpcIndexes`), seeds the epoch rows, and finalizes the store (`Watermark::Indexed`, then `meta` — a crash before `meta` leaves a store the next open wipes and re-inits). The node opens it in place instead of re-indexing the whole restored state; opt out with `--skip-grpc-indexes`. `init` and the restore share one indexing implementation (`GrpcLiveObjectRestorer`; `ParMakeLiveObjectIndexer` is lifetime-generic now). - **Chain-identity gate (`iota-snapshot`).** `verify_and_restore_epoch_info` rejects a snapshot whose manifest `chain_id` differs from this node's chain before writing any row. - **`RestoreEpochInfo` trait (`iota-snapshot`).** Separate single-method trait instead of a new `Restore` method: the two cover different snapshot payloads with different targets, so each call site requires exactly the capability it uses; the unified indexer (#11023) can implement both. - **No `epochs` migration (`iota-core`).** The deprecated `epochs` CF is dropped without migration: its rows lack the end-of-epoch fields, so they could never satisfy `EpochIndexed` and the backfill would overwrite them anyway. - **Open-epoch seeding (`iota-core`).** `initialize_current_epoch_info` keys off the open epoch (`open_epoch_of`): a restore lands on a closing checkpoint, and seeding that checkpoint's own (closed) epoch would leave the open epoch's row permanently missing and wedge the watermark. Its start checkpoint derives from `epoch_last_checkpoint_map` (new seq-only `CheckpointStore` accessor; the backwards scan over prunable summaries is deleted). - **Default snapshot source (`setups`).** Fullnode setups for mainnet/testnet/devnet ship a default `state-snapshot-read-config` pointing at the IOTA Foundation buckets; ignored unless the gRPC API is enabled and the index is incomplete. - **Misc.** Uploader honors `state_snapshot_write_config.concurrency` (default still 20); `get_latest_available_epoch` delegates to `iota_snapshot::reader::latest_available_epoch`; `snapshots.mdx` documents the restore flag and the backfill setup. ## Links to any relevant issues follow-up (PR-2) of #11453 · part of #11023 ## How the change has been tested - [x] Basic tests (linting, compilation, formatting, unit/integration tests) - [x] Patch-specific tests (correctness, functionality coverage) - [ ] I have added tests that prove my fix is effective or that my feature works - [ ] I have checked that new and existing unit tests pass locally with my changes --------- Co-authored-by: muXxer <git@muxxer.de>

# Description of change Some cleanup for #11697 - **Local epoch backfill: only missing data counts as pruned (`iota-core`, `iota-types`).** Real storage failures were swallowed as "data pruned, retry once a newer snapshot is published". Absent-data errors now carry `Kind::Missing` (was `custom`) and only those end the best-effort replay; anything else propagates. Nothing else reads the kind. - **Operator doc fixes (`iota-config`, `docs`).** The config doc still described the removed background task (the backfill is synchronous and gates startup); the snapshots guide now documents the refuse-to-start case when even the latest snapshot is older than the node's pruned-away history. ## How the change has been tested - [x] Basic tests (linting, compilation, formatting, unit/integration tests) - [ ] Patch-specific tests (correctness, functionality coverage) - [ ] I have added tests that prove my fix is effective or that my feature works - [ ] I have checked that new and existing unit tests pass locally with my changes