Skip to content

engine: branch merge stages detached chunk chains and publishes pins (RFC 0067 step 4) - #734

Closed
ragnorc wants to merge 3 commits into
claude/rfc-0067-ensure-indices-detachedfrom
claude/rfc-0067-merge-detached
Closed

ragnorc wants to merge 3 commits into
claude/rfc-0067-ensure-indices-detachedfrom
claude/rfc-0067-merge-detached

Conversation

@ragnorc

@ragnorc ragnorc commented Sep 16, 2026

Copy link
Copy Markdown
Contributor

RFC 0067 rollout step 4, third writer: branch merge stages detached commits. Stacked on #733 (ensure-indices), which is stacked on #731, #728, #727 and #725.

What changes

  • Every merged table's chunks chain as detached commits of the pin captured before classification. prepare_existing_merge_target opens the pin (open_pinned_for_write, promoting a pending predecessor first, staging behind a blocked one); each chunk of a proven insertion chain, an adopted delta or a three-way rewrite commits through commit_staged_detached on the previous chunk (MergeChain, one link per chunk within the existing MAX_BRANCH_MERGE_DATA_TRANSACTIONS ceiling, now checked by plan_merge_chain_length); the table's pin publishes expected + chain length with the tip as its staged version, in the one manifest commit. The writer then promotes every link in order (HeldPromotion gained a chain, and promote_held replays the links from the base, continuing from a twin a racing promoter landed).
  • No sidecar, no pre-minted identities, no in-process compensation. new_branch_merge_sidecar_v9 is cfg(test), confirm_branch_merge_sidecar_v9 and recover_failed_branch_merge_under_gates are deleted, the exact-commit outcome check and the RecoveryRequired wrapping are gone; a merge that fails before publication returns its error and leaves only detached chains and unreferenced forks, which cleanup classifies. A first-touch fork on a named target is created under the gates with no intent record. Pointer adoption copies the source's entry as it is, so a pending pin becomes the target's pin too.
  • Seams. branch_merge.post_sidecar_pre_fork, branch_merge.post_effects_pre_confirm and branch_merge.pre_error_recovery are gone; branch_merge.post_fork_pre_commit, branch_merge.post_table_effect and branch_merge.post_publish_pre_promotion are new; the chunk seams and post_authority_capture, post_candidate_validation, post_phase_b_pre_manifest_commit keep their names.
  • A first-touch fork accepts an inherited handle opened at a pending pin. The mutation path's pre-fork check compared the inherited handle's version with the manifest's published version; a handle opened at a pending pin resolves to the staged version, which the pointer-adoption scenario below exposed. The check now accepts the pin's staged version; the fork loop already promoted the pin before forking.
  • The classifier keeps the BranchMerge kind for sidecars written before the change, as engine: mutation and load stage detached commits and publish pins (RFC 0067 step 4a) #731 and engine: ensure_indices and the full-text rebuild stage detached commits (RFC 0067 step 4) #733 did for theirs; step 5 removes them together.

Tests

  • detached_commit_matrix.rs gains Writer::Merge (a branch holding the row under test, merged into main; the same windows, faults and recovery actors; a race with a main insert makes the merge lose its CAS).
  • failpoints.rs: twenty-five sidecar-era merge tests retire (phase-B recovery on reopen, pointer-slot replay, sidecar delete failure, rollback and restore, armed-prefix rollbacks, first-touch ref recovery, net-zero confirmation, the v1-sidecar classifier synthesis). New: every chunk window and the table and pre-publish windows leave no residue and the retry lands (assert_partial_merge_leaves_no_residue over the adopt and rewrite paths, plus the two multi-chunk fixtures); an interrupted promotion leaves a pending chain of two that the next write promotes link by link; a target advance after the detached effects loses the merge's CAS; a foreign linear commit after the effects blocks promotion, not the merge, and later writes chain behind it; a first-touch fork failure leaves garbage and the retry owns its own fork; pointer adoption carries a pending pin. The pre-arm fences (ABA, sync, source advance, late uncovered drift, pointer adoption) are unchanged. The two issue-694 live-compensation tests and the merge-sidecar body re-read test retire with the compensation path; the follower-attribution test only requires its Optimize producer to advance the table. The foreign-commit test documents that a conflict-free replay over a foreign commit may land as unreferenced history one past it while the pin still names the chain's tip.
  • The recovery-machinery tests (discovery races, the cleanup barrier, the storage-fault contracts, in-process healing, the S3 sidecar cell) take their pending sidecar from Optimize (graph_with_optimize_work + leave_confirmed_optimize_sidecar), the last writer that arms one that survives the next step.
  • write_cost.rs: a merge writes and deletes no control object; merge_cost.rs keeps its delta-scoped open count.
  • GQT: the issue-694 cases move to branch_merge.post_fork_pre_commit and branch_merge.post_table_effect; merge_armed_intent_blocks_main_until_reopen.gqt and the two runner tests that needed a corpus case with a genuine RecoveryRequired retire, because no corpus writer arms a recovery intent any more (the marker stays parsed and refused until the classifier retires); the runner fixtures rename the seam.
  • crates/omnigraph-dst: the crash-window catalog follows the renamed and new merge seams (the mutation and index renames land on engine: mutation and load stage detached commits and publish pins (RFC 0067 step 4a) #731 and engine: ensure_indices and the full-text rebuild stage detached commits (RFC 0067 step 4) #733).
  • Docs: writes.md (writer row and the merge's protocol), recovery.md, testing.md, the RFC 0045 example, release note.

Review regression tests (17 September 2026)

Seven failpoints.rs cases pin the boundaries the stack review reproduced, five of which failed before the fixes now in #727: the single-link pending-pin diff control, the blocked-chain and pending-merge diffs, cleanup on a blocked pin, the aged detached-manifest reaper, the reader racing a promotion, and a stale handle writing after another handle's promotion (the cause of the intermittent 409s in the server's concurrent-insert test, green eight of eight afterwards). The merge phase topology fix for the bench crate and the maintenance guide's blocked-pin sentence (branch merge and index maintenance stage behind a block) are here too.

…(RFC 0067 step 4)

Every merged table's chunks now commit as detached versions chained on
the pin captured before classification: the proven insertion chain, the
adopted delta and the three-way rewrite each land one link per chunk
within the existing transaction ceiling, the table's pin publishes the
linear base plus the chain length with the tip as its staged version, and
the writer promotes every link in order from the handles it holds. A
pointer adoption copies the source's entry as it is, so a pending pin
becomes the target's pin too; a first-touch fork on a named target is
created under the gates with no intent record.

No BranchMerge sidecar is armed, confirmed or deleted, and a merge that
fails before publication returns its error and leaves only reclaimable
garbage: the pre-minted identities, the exact-commit outcome check, the
in-process compensation and the RecoveryRequired wrapping are gone. The
mutation path's first-touch check accepts an inherited handle opened at a
pending pin's staged version, which pointer adoption exposed.

Seams branch_merge.post_sidecar_pre_fork, post_effects_pre_confirm and
pre_error_recovery are gone; post_fork_pre_commit, post_table_effect and
post_publish_pre_promotion name the new windows. The failure matrix gains
the merge writer, the sidecar-era merge tests retire in favour of
no-residue, pending-chain, CAS-loss, foreign-commit, first-touch and
pointer-adoption tests, the recovery-machinery tests take their sidecar
from Optimize, and the GQT known-failure fixture retires with the last
corpus writer that could arm a recovery intent. The classifier keeps the
BranchMerge kind until step 5.
…rs (RFC 0067)

The detached branch merge arms, confirms and deletes no sidecar, so the
`RecoveryArm`, `RecoveryConfirm` and `RecoveryCleanup` merge timing
phases record nothing. The bench's phase topology treated an unentered
`RecoveryArm` as a mismatch and failed three fixture merges in CI. The
three phases leave the engine's `MergeTimingPhase`, its probe accessors
and the rfc023 scenario, and the bench's stable and required phase
tables and stored-sample fixture; the merge guide's timing chain follows.
… (RFC 0067)

Seven `failpoints.rs` cases: the single-link pending-pin diff control, the
blocked-chain and pending-merge diffs, cleanup on a blocked pin, the aged
detached-manifest reaper, the reader racing a promotion, and a stale
handle writing after another handle's promotion. The five that reproduced
the review's defects failed before the fixes. The maintenance and write
guides describe the blocked-pin GC refusal and the age-gated reaper, and
the blocked-pin sentence now says branch merge and index maintenance stage
behind a block.
@ragnorc
ragnorc force-pushed the claude/rfc-0067-ensure-indices-detached branch from 74318a3 to 7bba37d Compare September 17, 2026 07:53
@ragnorc
ragnorc force-pushed the claude/rfc-0067-merge-detached branch from c122211 to f3fcc96 Compare September 17, 2026 07:53
@ragnorc

ragnorc commented Sep 17, 2026

Copy link
Copy Markdown
Contributor Author

Consolidated into #744, which carries this stack's complete history merged with current main and revalidated as one tree.

@ragnorc ragnorc closed this Sep 17, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant