perf(branches): preserve snapshots and reuse verified views

[aaltshuler]

What & why

Branch operations repeatedly reconstruct manifest history, and parent merges or cleanup can invalidate native table history still borrowed by a child. Preserve that history and reuse freshly verified source, target, and borrower views so repeated operations do less storage work while retaining the existing publication and recovery checks.

Stacked on #630, using codex/repro-untouched-table-merge as the base. That PR owns the original lagging-source-version merge fix. This PR contains the subsequent branch refactor and small-fixture diagnostics. Merge #630 first, then retarget this PR to main.

• Preserve owned target lineage and protect live borrowers during first-touch writes, recovery, cleanup, and deferred reclamation. Refuse an unowned target ref before arming recovery.
• Carry captured targets through merge without rebinding the shared handle. Separate strict inserts from known-present updates; bound candidate reads, transaction provenance reads, dependency checks, and reclamation concurrency.
• Reuse a source or borrower view only after fresh native-ref and full-incarnation validation. Share immutable lineage and a single deletion inventory; stale or unreadable authority takes the existing reconstruction/refusal path.
• Retain the projection computed from the successful publication attempt only when its base matches the coherent cached view, and install it only after lineage adoption. Avoid redundant manifest loads used solely to mint commit metadata.
• Add tiny fixtures varying retained history, retirement, cache/layout, live siblings, table width, ancestry, and supported legacy bare refs independently. Measure graph open, operation, first read, and reclamation separately.

Backing issue / RFC

• Refines the existing implementation of accepted RFC 0022: Unified graph-write protocol, including merge effect ownership, derived-state finalization, and §§7–8 branch control/maintenance. Preserves accepted RFC 0042: Incarnation-suffixed native branch refs.

Related performance work: #641 and #642. This PR does not close their broader acceptance criteria. Accepted-context reuse is part of canonical RFC 0057, accepted separately in #638. The included RFC 0058 document remains a draft proposal and is not implemented or used as authorization for a format change.

Measurements

The latest reuse comparison is edb8a87dbf31484d7261d025edf3822e4246e584 → 86aa508580097cf46fb4c32b55f053d41f98e412. Both saved release executables were built from verified clean sources. The reference already contains the earlier branch refactor below; these numbers isolate the later view-reuse changes.

Core matrix: 65 points / 195 verified samples per side, three repetitions, fixed rows=16, four-dimensional values, H0/H16/H64 history, cold/read-metadata-warm × compacted/uncompacted layouts. Largest fixture: 9.87 MiB, with maximum operation-process RSS about 65 MiB on each side.

Cold, uncompacted operation medians, excluding separately measured initial open:

Additional commits	Fork ms	Fork manifest reads	Delete acknowledgement ms	Delete manifest reads
0	4.00 → 3.38	15 → 13	5.78 → 5.63	25 → 22
16	5.97 → 3.47	31 → 13	11.97 → 10.77	88 → 69
64	10.42 → 3.71	79 → 13	26.60 → 22.36	280 → 213

At H64, fork removes one manifest scan and is 64% faster; deletion removes one scan and is 16% faster in this local matrix. Open, prewarm, first-read, and reclamation measurements did not cross the diagnostic regression threshold; first-read request counts were unchanged. Separate release width controls contain 12 points / 36 samples per side. Separate debug topology diagnostics cover ancestry, legacy refs, and concurrent merges; their timings are not pooled with release measurements.

Five original merge cells showed +11–21% timing increases with unchanged reads/scans. All original results were retained. A matched 84-sample alternating check did not reproduce those five flags (−5.5% to +4.7%); a further 36-sample fresh control measured 19.808 → 20.050 ms (+1.2%). No merge speedup or zero-overhead claim follows. These are local directional medians, not S3 measurements or tail-latency estimates; OS cache was uncontrolled.

The earlier refactor phase, measured separately at 24f5cb3c, reduced delete acknowledgement at 32 branches / 16 tables from 79.752 → 63.855 ms and total deletion from 93.343 → 73.213 ms. Its mixed-merge manifest GETs fell 1,261 → 1,049, while splitting insert/update effects raised table GETs 81 → 105 and PUTs 8 → 12. Stronger borrower proofs also raised that fixture's setup from 1.193 → 2.035 s and delete-process RSS from 48.984 → 54.938 MiB. These earlier results are not measurements of final HEAD and are not combined with the later baseline.

Reproduction and accounting controls are documented in benchmarks/README.md, scripts/bench-branch-age.py, and docs/dev/testing.md.

Checklist

• Changes concern branch snapshot ownership, redundant metadata work, and their measurement/correctness coverage.
• Existing branching, merge, recovery, maintenance, cache, cost, and source-guard owners extended; DST cost changes reviewed after repeat replay.
• Public/developer docs and release notes updated.
• Reviewed against architectural invariants: the manifest remains authoritative, publication is coherent, and missing derived views reconstruct safely.

Rebase and GQT verification — 2026-09-07

Rebased onto updated #630 at 47cb4c02; this stack now includes main 5f94a741 and Azim's branch statements/GQT support.

• Added parent_merge_preserves_lazy_child.gqt. It fails on fix(merge): preserve state when source table versions lag target #630 at 47cb4c02 after the parent's next write: the child's historical published dataset version 3 was reclaimed. The case records this witnessed red_on result.
• Added empty_merge_preserves_lazy_child_first_write.gqt. It already passed on fix(merge): preserve state when source table versions lag target #630 and guards the empty-delta path plus the child's first write from its original snapshot.
• Freshly rebuilt perf(branches): preserve snapshots and reuse verified views #662 runner at f57268dc: all 52 GQT cases passed, including fix(merge): preserve state when source table versions lag target #630's two regressions. New child cases take about 0.1 s each and seed only one row.
• Documentation, metadata, formatting and whitespace checks passed. Current-head CI passed the full workspace/failpoint test graph, AWS-server feature tests, both Clippy graphs, and the genuine V5/V6 format fence. The dedicated GQT and DST workflows also passed; DST ran 78 tests with 30 ignored instruments. Live RustFS/Azurite integration jobs were skipped by the PR workflow. Previous benchmark results below retain their original provenance.

The GQT runner keeps the new branch-control and per-step branch syntax while preserving this PR's boxed execution boundary.

Original implementation verification (before this rebase)

The following checks passed on 3a879157d99e4873b0fd8f5fc40034b8d871b249; the measured 86aa5085 added only 23 lines of testing documentation. Subsequent RFC-numbering corrections are documentation-only; focused documentation checks passed again. Formatting ran before only the reviewed DST cost-golden and documentation changes; Rust/build/test implementation was identical. No builds or benchmarks were repeated merely to open this PR.

• cargo test --workspace --locked --features omnigraph-engine/failpoints,omnigraph-cluster/failpoints --no-fail-fast -- --test-threads=1 — 2,963 top-level passed, 27 ignored, plus two nested helper passes. Includes GQT.
• From crates/omnigraph-dst, with its tokio_unstable configuration and OMNIGRAPH_DST_SEEDS=301: cargo test -p omnigraph-dst --locked --profile dst --no-fail-fast -- --test-threads=1 — 78 passed, 30 ignored. CPU/Rayon/Tokio pools 1, I/O 2. Optional concurrent and two-seed supplements also passed; the concurrent supplement reported nine scheduler escapes, so it does not establish strict replay.
• cargo clippy --workspace --all-targets --locked -- -D warnings -W clippy::dbg_macro — passed.
• cargo clippy --workspace --all-targets --locked --features omnigraph-engine/failpoints,omnigraph-cluster/failpoints -- -D warnings -W clippy::dbg_macro — passed.
• cargo test --workspace --locked --features omnigraph-engine/failpoints,omnigraph-cluster/failpoints,omnigraph-server/aws --lib --bins --test auth_policy --test boot_settings --test data_routes --test multi_graph --test openapi --test s3 --test schema_routes --test stored_queries --no-fail-fast -- --test-threads=1 — 1,827 top-level passed, 4 ignored, plus two nested helpers; includes all eight server integration owners. Live S3/Azure environments were unconfigured, so this verifies the feature graph, not live cloud operation.
• cargo fmt --all --check, git diff --check, bash scripts/check-agents-md.sh, python3 scripts/check-docs.py, and python3 scripts/check-workflow-action-pins.py — passed.

Notes for reviewers

Review native-ref ownership and recovery first, then exact-view capture and projection handoff, then the bounded read paths and benchmark controls. Projection outcomes are boxed to avoid enlarging async stack frames; a small boxed GQT runner boundary fixes a separately reproduced default-stack failure.

No new persistent state, storage/recovery format, or dependency upgrade is introduced. Publisher validation still scans history; uncached create-from and cold opens remain history-dependent. Eight-sibling deletion still requires nine scans in the width control. Same-graph merges still serialize. Retained projections can consume memory proportional to historical table lifetimes. RFCs 0024/0025/0058, historical ancestry retention, and broader cold-access/concurrency work remain separate.