detached table commits replace recovery sidecars - #744
Merged
Merged
Conversation
…record, with Lance fences RFC 0067 replaces the recovery sidecars with detached table commits, publication of (target, staged, uuid) pins and promotion by transaction replay; RFC 0068 is its sequel, one immutable commit record per graph commit. Both are drafts with their probe, prototype, S3, cross-process and failure-matrix evidence recorded; the prototype itself lives on the spike/rfc-0066-detached-prototype branch and is never merged. The four substrate behaviours the design depends on become lance_surface_guards fences: a stale linear commit rebases over a moved HEAD at zero retries, a detached commit is private and chains, replay lands an identical twin, and a replay over its own twin is refused only for self-conflicting transaction kinds.
Readers, the promoter and cleanup learn to handle a table pin that names a detached Lance version, without any writer producing one yet. On existing graphs the change is inert: no pin carries the new fields. - TableVersionMetadata carries staged_version and transaction_uuid in the manifest row and the namespace request metadata map. - open_pinned_dataset resolves a pin to its linear target, its staged version while the target is pending or foreign, or reclaimed once the linear head has passed an absent target; both dataset-open paths use it. - The Blob facade and the change-feed witness accept the transaction uuid as identity; the Blob version check accepts a pending pin's staged version. - A version's transaction identity is read from the transaction file name Lance records in the manifest, so no request beyond the manifest is needed; the existing twin fence pins the name. - TableStorage gains commit_staged_detached, promote_detached and transaction_identity; the registry pins their call sites. - db/omnigraph/promotion.rs promotes one pin: target first, chains through read-version links, tolerates reaped predecessors, reports foreign occupants as blocked. - Cleanup promotes pending pins inside its per-branch loop before any reclamation and deletes promoted pins' detached manifests. - Seam promotion.pre_replay for the race and crash matrix of step 2.
… 0067 review found Four defects reproduced against the stack, plus one the server's concurrent insert test exposed once the writers landed: - Cleanup skipped nothing when a pin's promotion was blocked: HEAD equalled the pin's target numerically, so stock version GC ran and could delete the detached version's data files once older than Lance's unverified threshold. Cleanup now records a blocked pin's table, refuses version GC for it in that table's stats row, and protects every link of the blocked chain from the reaper. - Cleanup deleted only the current tip pin's detached manifest, so superseded pins, promoted chain links and attempts that never published accumulated forever. With `--older-than`, cleanup now lists each table location's `_versions/` and reaps every detached manifest older than the threshold that no pending chain protects; without an age policy it reaps none, since a recent manifest may belong to a write in flight. - The snapshot diff filtered `_row_last_updated_at_version` against the linear interval even when the `to` endpoint resolved to a pending pin's detached version, whose rows staged on a detached predecessor carry stamps outside that interval. The same-lineage diff now hands a detached endpoint to the exact ID/content comparison. - The pinned resolver reported reclaimed history when a promotion landed between its target probe and its HEAD read. It now re-reads the exact target once HEAD has reached it, and reports reclaimed history only when the target is still absent. - A writer whose captured snapshot predated another writer's promotion saw the linear HEAD one past its published version and was refused with the non-retryable "run `omnigraph repair`" conflict; concurrent `/change` inserts returned 409 in about half of the runs. The baseline check now reads the current manifest first and returns `ReadSetChanged` when it explains the HEAD, so the writer reprepares; genuinely foreign drift is still refused.
…t route from v8 and v9 Normal open serves internal manifest schema v10 only. v10 lets a table registration name a detached Lance version whose linear target is published before it exists; an older binary would open the absent target and misread a pending pin as reclaimed history, so the stamp refuses it before any open. Both system column vintages are born at v10 and the vintage is read from the schema IR, never from the stamp. - omnigraph upgrade defaults to v10: v6 -> v7 -> v8 -> v10, v7 -> v8 -> v10, v8 -> v10 and v9 -> v10. The new detached-pins-v8-v9-to-v10 handler restamps main and every live branch under one intent and touches nothing else; retired ancestors keep their stamp; a receipt from an earlier step of the same route counts as a completed predecessor. v9 is not a target. - The system-column respelling leaves the storage route and runs on a served graph with no stamp change; the dead per-vintage stamp floor is removed. - Refusals name the in-place route for v6 to v9 and the 0.11.x release for v8 and v9. - Tests, CI job names, the OpenAPI description, and the versioning, upgrade, testing, CI and Lance guides follow; release note under compatibility.
The workflow greps already name the current fence tests after the served format, and the default route now continues through a branched v8 graph to v10 and refuses an ambiguous lifetime at its v7 -> v8 step: the genuine v0.9/v0.10 journeys check first, execute, then prove the v8 stop is a refused downgrade.
…C 0067 step 4a) Every mutation and load table effect is a detached commit of its pinned base: no recovery sidecar is armed, confirmed or deleted, a table's linear HEAD never moves before publication, and the one manifest CAS publishes each pin as (base + 1, staged version, transaction uuid). The writer then promotes each pin from the handles it holds, best effort; a pending pin is served through its staged version and promoted by the next writer of the table or by cleanup. Writers open the pin, never HEAD, promoting a pending predecessor first. A pin blocked by a foreign linear commit stages the next write from the detached version, whose promotion waits behind the block. The writers that still commit on the linear HEAD (branch merge, index maintenance, schema apply, the system-column upgrade, Optimize, repair) promote a pending pin before they plan and refuse a blocked one; repair reports such a table as blocked_promotion and never adopts the foreign commit. Seams mutation.post_arm_pre_effect, mutation.post_sidecar_pre_fork, mutation.sidecar_confirm_put and mutation.sidecar_post_publish_delete are gone; mutation.post_publish_pre_promotion, promotion.post_landed and cleanup.pre_reap name the new windows. The detached commit matrix, the bridge and blocked-pin failpoint tests, the cost instruments and the GQT corpus (a merge intent as the known-failure case) own the evidence.
…maining sidecar lifecycle CI's RustFS shard filters the failpoint suite by s3_, and the load test it matched armed a mutation sidecar that RFC 0067 removed. Two cells replace it: a load interrupted after publication leaves a pending pin on S3 that the next load promotes, and the index writer's sidecar (until its own conversion) is put, listed, rolled forward and deleted through the S3 adapter by the next write.
The spelling check reads DELETEs as a typo.
…car seam mutation.post_sidecar_pre_fork is gone; its slot names mutation.post_publish_pre_promotion so the catalog's index-derived seeds stay stable, and the harness recipes that keyed on the fork window keep only mutation.post_fork_pre_commit.
The cost golden loses every mutation-class sidecar PUT, DELETE and ref LIST and gains one Lance manifest PUT per touched table for the twin; Cleanup reaps the detached manifests. Four pins whose shape needs a mutation sidecar retire, the issue-554 panel keeps its availability guarantee without requiring a deferred-recovery row, and the bounded-staleness and ack-loss pins take new seeds from two search instruments, the latter expecting exactly-once application under client retry.
The spelling check reads DELETEs as a typo.
…ts (RFC 0067 step 4) The index writer follows mutation and load: it opens each productive table at its pin, stages the complete index batch before the gates as before, commits every batch as a detached version of the pin, publishes the pins in the one manifest commit and promotes them from the handles it holds. No EnsureIndices sidecar is armed, confirmed or deleted; a first-touch fork on a branch is created under the gates with no intent record, and the full-text certificate written before the commit serves search through a pending pin. Seams ensure_indices.post_sidecar_pre_fork and ensure_indices.post_effects_pre_confirm are gone; ensure_indices.post_fork_pre_commit and ensure_indices.post_publish_pre_promotion name the new windows. The failure matrix gains the index writer, the sidecar-era index tests retire, and the recovery-machinery tests take their pending sidecar from branch merge. The classifier keeps the EnsureIndices kind until step 5.
… seams ensure_indices.post_sidecar_pre_fork and post_effects_pre_confirm are gone; their slots name post_fork_pre_commit and post_publish_pre_promotion, the harness recipes follow, and the maintenance-obligation cell crashes the index writer at post_table_effect, the window that still leaves work.
The cost golden follows the index writer's pinned opens and promotions, and the bounded-staleness pin takes a seed that stays green on the detached mutation and index schedules.
…(RFC 0067 step 4) Every merged table's chunks now commit as detached versions chained on the pin captured before classification: the proven insertion chain, the adopted delta and the three-way rewrite each land one link per chunk within the existing transaction ceiling, the table's pin publishes the linear base plus the chain length with the tip as its staged version, and the writer promotes every link in order from the handles it holds. A pointer adoption copies the source's entry as it is, so a pending pin becomes the target's pin too; a first-touch fork on a named target is created under the gates with no intent record. No BranchMerge sidecar is armed, confirmed or deleted, and a merge that fails before publication returns its error and leaves only reclaimable garbage: the pre-minted identities, the exact-commit outcome check, the in-process compensation and the RecoveryRequired wrapping are gone. The mutation path's first-touch check accepts an inherited handle opened at a pending pin's staged version, which pointer adoption exposed. Seams branch_merge.post_sidecar_pre_fork, post_effects_pre_confirm and pre_error_recovery are gone; post_fork_pre_commit, post_table_effect and post_publish_pre_promotion name the new windows. The failure matrix gains the merge writer, the sidecar-era merge tests retire in favour of no-residue, pending-chain, CAS-loss, foreign-commit, first-touch and pointer-adoption tests, the recovery-machinery tests take their sidecar from Optimize, and the GQT known-failure fixture retires with the last corpus writer that could arm a recovery intent. The classifier keeps the BranchMerge kind until step 5.
…rs (RFC 0067) The detached branch merge arms, confirms and deletes no sidecar, so the `RecoveryArm`, `RecoveryConfirm` and `RecoveryCleanup` merge timing phases record nothing. The bench's phase topology treated an unentered `RecoveryArm` as a mismatch and failed three fixture merges in CI. The three phases leave the engine's `MergeTimingPhase`, its probe accessors and the rfc023 scenario, and the bench's stable and required phase tables and stored-sample fixture; the merge guide's timing chain follows.
… (RFC 0067) Seven `failpoints.rs` cases: the single-link pending-pin diff control, the blocked-chain and pending-merge diffs, cleanup on a blocked pin, the aged detached-manifest reaper, the reader racing a promotion, and a stale handle writing after another handle's promotion. The five that reproduced the review's defects failed before the fixes. The maintenance and write guides describe the blocked-pin GC refusal and the age-gated reaper, and the blocked-pin sentence now says branch merge and index maintenance stage behind a block.
…act without a sidecar (RFC 0067 step 4) Schema apply is the fourth writer moved to detached table commits. An existing-table rewrite is a detached Overwrite of the promoted HEAD, published as a pin one past the published version and promoted after the manifest commit. An added type stays a linear version-one create at its identity path; that path is a deterministic function of the accepted identity allocator (the RFC's "fresh incarnation path" was wrong), so an attempt that died after creating the dataset left it exactly where the retry creates it, and the retry reclaims the unregistered leftover under the schema sentinel instead of refusing to claim unowned physical state. The schema contract is staged with the graph commit that publishes it recorded in `__schema_state.json.staging`, and the writer installs the live contract from memory after the commit. No SchemaApply sidecar is written: a failure before the manifest commit returns the plain error and leaves detached versions, a created dataset and a staged contract that the next read-write open discards; a failure after it reports `RecoveryRequired` naming the published commit, and the next read-write open, or the same handle's next write, installs the contract because that commit is in main's lineage. A read-only open refuses a published but uninstalled contract and serves an unpublished staging as absent. The open also reclaims a sentinel left by a crashed apply, which previously stayed behind whenever an apply died after deleting its sidecar, and the release is idempotent. The RFC 0040 system-column upgrade keeps the v9 exact protocol, its unmarked staging and its sidecar; the schema-apply sidecar constructor is test-only until step 5 removes the classifier. Lance fence: an Overwrite replayed over its own twin is idempotent (Lance recognises the committed transaction and adds no version), so racing promoters of a schema-apply pin leave no residue. Tests: the schema-apply failpoint cells become no-residue, staging discard, staging promotion, leftover reclaim, same-handle heal and manifest-CAS-loss tests; the sidecar-rollback, foreign-winner and Optimize late-sidecar tests whose producer no longer exists are retired; `detached_commit_matrix` gains the `SchemaApply` writer (52 cells in the full run); `write_cost` gains `schema_apply_writes_no_control_object`; the seam `schema_apply.post_publish_pre_promotion` is new and the DST catalog lists 72 windows.
…shes with an exact pin CAS (RFC 0067 step 4) Optimize no longer arms a maintenance sidecar, commits on a table's linear HEAD, or publishes monotonically over whatever pin is current. Each productive table's work is a chain of detached commits of its pin: one Rewrite built from Lance's compaction plan executed against the pinned base, with fragment ids allocated above the base's high-water mark so no ReserveFragments is needed; a whole rebuild of every foldable index whose coverage lags the rewritten layout, under its name and keeping a vector index's partition count, since Lance 11 folds only through a linear commit and keeps merge_indices crate-private; and a build of every declared index the table lacks. The batch publishes once through the shared publisher with an exact CAS on every pin it planned from and the captured graph head, then promotes the held chains. A pin a concurrent writer moved fails the run with a read-set conflict and the next run re-plans; a failure before the manifest commit leaves only reclaimable detached versions; a failure after it leaves pending pins the next writer of each table or cleanup promotes. Optimize no longer strips a stale lance.auto_cleanup configuration from data tables: every engine commit and every promotion replay skips Lance's auto-cleanup, so the key is inert, and a delete-only UpdateConfig replay would not conflict with its twin. The internal __manifest compaction keeps its linear compact_files and strip. Seams: optimize.post_compact_pre_reindex is retired; optimize.post_table_effect (after each detached link) and optimize.post_publish_pre_promotion are added. The DST catalog lists 73 windows and the cost golden records the new operation counts. Two DST pins retire as vacuous by construction, each after its search instrument found no schedule entering the shape: the ack-loss client-retry pin (the sidecar was the standard workload's only adapter-realm write whose lost acknowledgement failed an op; the failpoint suite owns the manifest-CAS contract) and the keep-serving arbitration-widening pin (no workload writer arms a recovery operation a live handle could be deferred on). Tests: failpoints.rs converts the six Optimize cells to the detached contract (no residue before publication, pending pins after a lost acknowledgement or a skipped promotion, no pin for a pending-only table, partial batch leaves no residue, the two branch-gate cells park on a rendezvous, and the strict delete queued behind Optimize reports the typed read-set conflict its caller retries) and retires the ten recovery-machinery tests and the S3 sidecar-lifecycle cell whose only producer was the Optimize sidecar, replacing the latter with an Optimize pending-pin S3 cell. detached_commit_matrix.rs gains Writer::Optimize (four windows, three faults). write_cost.rs pins that Optimize writes and deletes no control object. maintenance.rs keeps the stale-config cell as a versions-preserved assertion. search.rs builds the issue-567 partitioned IVF fixture explicitly, because the engine's own vector builds are one-partition and a fold keeps the partition count. forbidden_apis.rs registers the two staged primitives and the corrected counts. Docs: writes.md, recovery.md, the maintenance guide, the release notes and the RFC 0067 rows (no ReserveFragments, no config strip, the fold is a detached whole rebuild, merge_indices stays crate-private).
… sidecar (RFC 0067 step 5) The RFC 0040 system-column upgrade was the last writer that armed a recovery sidecar. It now has schema apply's RFC 0067 shape: every table's rename-only Project commits detached from its promoted pin, the staged schema contract names the graph commit that publishes it, one manifest CAS publishes every pin under the exact graph head, the contract installs from memory, and the held renames promote. Lance refuses a Project replayed over its own twin (check_project_txn), so racing promoters leave nothing; the twin-replay fence gains that case. Since v10 both column vintages share one __manifest stamp, so the stamp advance the old protocol carried was already a no-op; it is gone, and the vintage is the contract's system-columns feature alone. A failure before the manifest commit returns the plain error and leaves the graph unchanged: the detached renames are reclaimable garbage, the next read-write open discards the unpublished staging and reclaims a dead writer's sentinel, and a read-only open serves the legacy contract. A failure after it reports RecoveryRequired naming the published commit; the next read-write open, or the same handle's next write entry, installs the contract, and a read-only open refuses until then. With the promotion skipped the renamed tables stay pending pins the next writer or cleanup promotes. Seams: schema_apply.post_sidecar_pre_effect is renamed schema_apply.post_lock_pre_effect (its only remaining meaning, shared with schema apply); system_column_upgrade.after_stamp_advance is gone. Tests: system_column_upgrade.rs converts the six roll-forward cells into four no-residue windows, the post-commit cell finished by the next open, the same-handle heal, the skipped-promotion cell, the same-handle retry and the dead writer's sentinel reclaimed at open, and pins that the upgrade writes six contract files, deletes three staging files and no sidecar; the two recovery-executor cells retire with their producer. The registry rows follow. write_sidecar and the system-column sidecar constructor are test-only until the classifier leaves in the next change. Docs: recovery.md, testing.md, lance.md, the upgrade guide, the release notes, and the RFC 0067 and RFC 0040 rows.
Every writer now stages detached table commits and publishes pins, so nothing arms a recovery sidecar and nothing is left for a classifier to decide. This removes the machinery: - db/manifest/recovery.rs (the sidecar grammar, shape validators, effect classifier, roll-forward and rollback executors, Restore compensation, both recovery modes and their in-source truth tables) and db/recovery_audit.rs (the recovery audit dataset, which has no appender left) are deleted. - The write-entry heal and barrier are gone: heal_pending_recovery_sidecars and its for-write and for-branch-delete variants, the under-gate rechecks, the staging owner check, Optimize's and cleanup's clean-recovery-state refusals, repair's sidecar probe, and the sidecar arm of the baseline check. What remains of the write-entry pass is settle_pending_schema_install: a flag-gated install of this handle's own published schema contract, with no storage probe on the common path. - A read-write open no longer sweeps: it refuses a graph whose __recovery/ directory still holds a sidecar (db/manifest/legacy_sidecars.rs; only a build that predates detached commits can have written one, and this build cannot interpret it), settles a staged schema contract by its publishing commit, and reclaims a stale schema-apply sentinel. The storage upgrade keeps the same refusal. A read-only open never lists __recovery/ and keeps only the published-contract coherence check, now in schema_state.rs. refresh installs a published staged contract and nothing else. - Seams recovery.*, optimize.post_recovery_check_pre_main_gate and system_column_upgrade.after_lock_reclaim are gone; branch_control.post_recovery_barrier is renamed branch_control.pre_gates and cleanup.post_recovery_check_pre_gates is renamed cleanup.pre_gates (the one window where a failure aborts a whole cleanup run; per-table GC failures are isolated). The DST crash-window catalog lists 61 windows. - Dead code the removal exposed goes with it: publish_stamp_advance, the unmarked-staging writer and its exact-staging helpers, CommitGraph::latest_commit_matching, TableStore::append_batch, the fork helper's unused operation-id parameter, the two Lance restore fences the recovery sweep depended on, and the retired audit table's name in the internal-table list; the coordinator's open and open_branch and the publisher's publish shorthand are test-only. Merge's limit and ownership messages say "detached transaction chain" and "planned bound" instead of "recovery transactions" and "armed bound". repair's drift adoption stays. It is not sidecar machinery, and it is the operator's one remedy for a foreign linear commit that arrived before any pending pin; how a blocked table is unblocked remains the RFC's open question. The GQ logic test format loses its known_failure marker, which existed only to pin known recovery-sidecar defects and was carried by no case. Tests: tests/recovery.rs shrinks to the contracts that remain (a clean open creates nothing, a legacy sidecar refuses a read-write open and not a read-only one, index maintenance leaves no sidecar, a read-only open never touches schema staging); the failpoint and maintenance cells that injected synthetic sidecars retire; the recovery test helpers shrink to the two with users; the storage-upgrade refusal uses a plain file; the registry drops the recovery executor rows. DST: the double-fault lever dies on the open path instead of an unreachable recovery seam; the harness no longer reads a recovery audit; the staleness pin moves to seed 278 because removing the per-write __recovery/ listing shifted the storage-action schedule; the sidecar-weather pin retires because lost and misdirected writes only ever acted on sidecar writes; the cost golden loses every per-operation adapter listing. Docs: recovery.md is rewritten around pins, promotion, staged contracts and the legacy-sidecar refusal; writes.md, invariants.md (invariants 2 and 5 and the one-mutation-process boundary), AGENTS.md rule 4, versioning.md, testing.md, merge.md, lance.md, the user troubleshooting and maintenance guides, the release notes, RFC 0067's step 5, a removal note in RFC 0045, and the engine's own comments, which no longer describe sidecars. The OpenAPI description of the resource-limit output drops its sidecar wording.
The keep-serving watch existed to prove issue #554: a live handle refused on its own failed attempt's recovery sidecar, with reconcile's reopen deferred so the wedge stayed observable. Since RFC 0067 no writer arms a recovery operation, so the shape cannot be entered: the widened-arbitration pin and its seed search retired earlier after their searches found no schedule reaching it, and this removes the machinery itself. Removed from the harness: Scenario::keep_serving_ops, KeepServingWatch, WatchInterrupt, WatchRuling, the composition-hypothesis widening (op_modes, CompositionHypothesis, composition_hypotheses, reconcile_watch_resolution), resolve_keep_serving_watch, the KEEP_SERVING row prefixes, the six arming and resolution sites in the universe loop, and the LiveWriteAvailability oracle (the census records 21 oracles, 24 detectors). The clean-recovery-state legality carve-out and its reopen-heals-barrier row go with it: the engine no longer emits that text. reconcile_after_failure and its single-op arbitration are unchanged. The issue-554 panel stays as dst_availability_panel_survives_harsh_faults_issue_554: the same 13 seeds under the same harsh fault plan, every seed must run green; any red is a genuine availability or oracle regression rather than a designed wedge. The cost golden is unchanged, which is the knob's standing constraint (it never perturbed a pinned universe's op stream or draws) holding through its removal.
This was referenced Sep 17, 2026
engine: the system-column upgrade stages detached renames and arms no sidecar (RFC 0067 step 5)
#740
Closed
This was referenced Sep 17, 2026
Closed
Closed
…hed-table-commits # Conflicts: # crates/omnigraph-dst/src/harness.rs # crates/omnigraph-gqt/src/dst_runner.rs # crates/omnigraph-gqt/src/dst_runner/known_failure.rs # crates/omnigraph-gqt/src/dst_runner/known_failure/tests.rs # crates/omnigraph-gqt/src/runner_config.rs # crates/omnigraph-gqt/tests/runner_dispatch.rs # crates/omnigraph/src/db/manifest.rs # crates/omnigraph/src/db/manifest/recovery.rs # crates/omnigraph/src/exec/staging.rs # crates/omnigraph/src/seams.rs # crates/omnigraph/tests/failpoints.rs # crates/omnigraph/tests/forbidden_apis.rs # crates/omnigraph/tests/helpers/recovery.rs # crates/omnigraph/tests/recovery.rs # crates/omnigraph/tests/search.rs # crates/omnigraph/tests/system_column_upgrade.rs # crates/omnigraph/tests/warm_read_cost.rs # docs/dev/ci.md # docs/dev/recovery.md # docs/dev/testing.md # docs/releases/v0.12.0.md # docs/rfcs/README.md
ragnorc
force-pushed
the
claude/rfc-0067-detached-table-commits
branch
from
September 17, 2026 21:59
dea9aec to
d76ffab
Compare
DST: the Lance realm gains the ack-loss verb (acknowledgement lost AFTER the store applied a write - put, copy, per-item delete - which includes the __manifest commit puts, the graph-publication door); the workload's client retries converge against their own durable-but-denied commits and every oracle holds. The quarantined schema-apply slot returns as the schema_ops face: monotone additive applies under seeded schedules, the engine's mono-branch refusal modeled as a predicted legal rejection, the schema_apply crash windows exercised (the post-publish window asserted absorbing per RFC 0067), and the persisted-write lie verbs revived against the schema control objects with a detected-or-harmless canary (the engine answers a misdirected __schema_state.json install with a loud typed refusal). The hunt, reach, predict-triage and fleet instruments enable the face for the schema families; pinned plans keep their exact draw sequences. detached_commit_matrix: three new writers - Load (two-row JSONL append through the loader's staging), FtsRebuild (explicit full-text rebuild on the ensure-indices seams), and SystemColumnUpgrade (RFC 0040 on a graph born legacy, through the schema-apply seams, with a check-outcome vintage oracle and sentinel-refused racers). 246 cells green. RFC 0067's DST validation gate is recorded closed; testing.md's matrix row names the ten writers.
… review Correctness/robustness: - cleanup: add a minimum detached-manifest retention floor so `cleanup --older-than 0` can never reap a cross-process writer's staged-but-unpublished manifest and brick its pin on a crash. That manifest is absent from protected_detached (published pins only), so the age floor is its sole guard. Regression test strands one and proves it survives zero-age cleanup, then is reaped once aged. - publisher: resolve an ambiguous manifest publish. When the __manifest CAS lands durably but its acknowledgement is lost, read the stable lineage commit id back and return the success outcome the attempt would have produced, rather than an opaque failure a non-idempotent retry could double-apply; only a genuinely unconfirmable outcome surfaces as a typed in-doubt error. New `publish.post_merge_pre_ack` crash-window seam and a regression test drive the durable-but-lost-ack path. - open_pinned_dataset: the latest<target arm now re-checks the target on a reclaimed staged read, matching the latest>=target arm, so a concurrent cleanup promoting-and-reaping the tip no longer yields a spurious HistoricalVersionReclaimed. - walk_chain: bound the read-version walk so a corrupt/cyclic manifest fails loud instead of hanging every writer. - promote_detached: reject Append/ReserveFragments/UpdateConfig/Restore/Clone up front, not just Append, so a corrupt pin cannot land a stray commit before the post-commit backstop catches it. - schema apply: stop swallowing a failed sentinel release on the mono-branch refusal path; warn so a leaked sentinel is observable. Test soundness (tightening this session's own additions): - drop the near-vacuous schema-weather refusal match, require the engine's exact control-plane string; - assert the client-retry path actually fires in the Lance ack-loss pin; - assert the full-text rebuild matrix cells name a real rebuilt index. Docs: remove the stale 0.11-cycle sidecar-heal and removed-GQT-seam bullets from the v0.12 notes and describe the lost-ack semantics; caveat the invariants 'does not tear' claim; fix the RFC 0045 known-failure ordering line, the writes.md table row, the RFC 0067 front matter and registry row, and the maintenance blocked-promotion list; drop dead code and stale recovery_audit/keep-serving references; regenerate the detector census.
ragnorc
changed the title
… 0067) The retired checks watched one wedge class: a live handle refused with RecoveryRequired on its own failed attempt's sidecar. The class is gone; the contract it enforced is general and now has owners at every tier: after any failed operation on a live handle, once the fault source stops, the same Omnigraph instance's next write succeeds without reopening. Engine: a schema-apply/upgrade sentinel this handle failed to release is retried at the next write entry (note_failed_sentinel_release, folded into settle_pending_schema_install ahead of the sentinel gate), so a transient release fault no longer wedges the handle until reopen. Matrix: the same-handle recovery actor joins the DEFAULT run (Return cells of every writer x window; 287 cells), no longer gated behind OMNIGRAPH_MATRIX=full. Failpoints: three liveness owners — persistent pre-publish faults stop and the handle writes again with nothing resurrected; persistent lost acknowledgements stop and every denied-but-durable row is present exactly once; and a write-family seam sweep that fails each non-matrix write seam once on one live handle and proves the next ordinary write succeeds, including schema_apply.after_manifest_commit, where the same handle heals its own published-but-uninstalled contract inline. DST: Scenario::keep_handle judges every failed op's aftermath on the SAME live handle instead of reopening (crash knobs refused; the report counts reopens), and the pinned dst_fault_storm_on_one_live_handle_keeps_writing runs an adapter-realm storm with lost acks and client retries on one never-reopened handle — zero reopens, strict replay, oracles hold. recovery.md gains the liveness contract; testing.md and the release notes follow.
The instrument RFC 0067's 'What to measure before promising numbers' section asks for: sustained commits/sec per branch under N closed-loop writers over clones of one Session (the production server shape), on the manifest-shape and backend axes the RFC names (compact vs fragmented via the existing age/layout controls; local fs vs an s3:// target with a reachability refusal and unique-prefix teardown). Honest by construction, pinned by a new contract test: the record self-labels driver: closed-loop and claim_grade: false (RFC 0039 Rule 1 — coordinated omission), latencies are service_time_* only; the counting reopen and every spawned worker run under one task-local QueryIoProbes value plus a CountingStorageAdapter for the control plane, with per-op divisions (manifest_writes_per_commit is where a future group-commit change must fall below one); the measured window opens after a warmup drain; verification runs on a fresh handle after every counter and clock is read, checks exact per-branch row counts and sampled keyed read-backs, and surfaces read-back query errors instead of swallowing them. A typed ReadSetChanged authority conflict (the graph head moved under a concurrent writer) is retried by the driver like a real client inside the same op's service time and counted as authority_conflicts; any other worker error, key conflict, or verification mismatch fails the run. A watchdog bounds a wedged remote store; records carry an attestation block (engine features, RUSTFLAGS, tokio_unstable cfg, worker count). Smoke evidence (local fs, 16 seeded rows): 2 writers sustain ~28 commits/sec with p95 267ms; 8 writers sustain ~26 commits/sec with p95 1.40s and 9 retried authority conflicts, all acknowledged writes exact on verification — the flat-rate, rising-percentile serialization baseline the RFC's throughput path targets. Diagnostic evidence only; records do not enter the durable archive.
The names guard forbids literal "branch_merge." seam strings in failpoints.rs (a typo'd literal would silently never fire); the liveness sweep's two merge cells now use the compile-checked catalog statics like the rest of the suite. Guard and sweep green.
Integrate current main while retaining detached-commit semantics. Require published-pin and UUID-matching twin proof for reclamation, retain uncertain staging and its data, resolve lost acknowledgements against the attempted manifest, and isolate blocked-chain cleanup per table. Extend the existing failpoints harness and update maintenance documentation and integration guards. Validated with the full workspace and GQT suites, full detached-commit matrix, Lance guards, maintenance/recovery owners, clippy, formatting, and documentation/dependency checks.
azimafroozeh
approved these changes
Sep 21, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
RFC 0067 — detached table commits — implemented end to end, in one PR. This consolidates the stacked draft PRs #725, #727, #728, #731, #733, #734, #735, #739, #740, #741 and #743 (each closed with a pointer here), merged with current
main(the sessions feature, store places, path-gated CI) and revalidated as one tree.What this PR does
Every graph writer used to coordinate multi-table crash safety through recovery sidecars: a durable intent file under
__recovery/, linear Lance commits behind it, and a reopen-time classifier that rolled effects forward or back. This PR replaces that protocol and then deletes it.__manifestCAS as a pin(target version, staged version, transaction uuid)plus lineage, then promoted onto the linear history by replaying the recorded transaction (its "twin"). A failure before the CAS leaves only unreferenced garbage the nextcleanupreaps; a failure after it leaves a pending pin that reads serve through the staged version and the next writer of the table, orcleanup, promotes. Lance's own conflict rules make promotion race-safe: every staged kind is refused when replayed over its own twin (fenced inlance_surface_guards).ensure_indicesand the full-text rebuild, branch merge (chunk chains, pointer adoption, first-touch forks without intent records), schema apply (staged contract bound to its publishing commit), Optimize (detached compaction rewrite, whole-rebuild index folds keeping a vector index's partition count, exact pin CAS instead of the monotonic publish), and the RFC 0040 system-column upgrade (detached rename-only projections, no stamp advance since both vintages share v10).omnigraph upgraderoutes v8/v9 to v10; the V5↔V10 and v6↔v10 fences and the storage-upgrade CI journeys cover it.Restorecompensation, the recovery audit table, the write-entry heal and barrier, the per-write__recovery/listing, the GQTknown_failuremarker, and the DST keep-serving machinery. A read-write open and the storage upgrade refuse a graph still carrying a sidecar from an older build; a read-only open never looks.repairkeeps its drift adoption (the one remedy for a foreign commit that lands before any pin exists) and gainsblocked_promotionreporting.optimizecan leave a pending Mutation recovery barrier that a long-lived server never clears #554 (a directoptimizebeside a live server wedged writers), bug: foreign-named sidecar in __recovery/ is never removed by recovery, permanently blocking writes on its branch #601 (mis-named sidecars) and Transient 500 'storage read failed …/__recovery/<ulid>.json not found' under concurrent writes (v0.7.0) #330 (sidecar read racing its own cleanup). Writers no longer pause each other through recovery barriers; a strict update or delete that loses to a concurrent publication gets the typed read-set conflict and retries.Where the intent lives, and crash semantics
The old protocol stored intent first — a durable sidecar saying "here is exactly what I plan to commit" — and then needed a classifier to reconcile plan against reality after a crash. This protocol arranges things so intent never exists separately from outcome:
cleanupreaps the leftovers. There is no record saying "finish this", because nothing must be finished.(target linear version, staged detached version, transaction uuid)is the intent record, published in the same atomic write as the outcome it describes — the two can never disagree.lance_surface_guards).The contract this yields: an acknowledged write is never lost; an unacknowledged write is never resurrected. Acknowledgement is sent only after the CAS is durable, so a crash before it loses only unacknowledged staging effort — the same outcome a WAL database reaches by logging intent and then rolling it back (presumed abort). The old sweep could do the opposite: roll forward and publish a write whose client had already been told it failed — heuristic-commit behavior, and the source of the ghost-write classes the DST arbitration existed to reason about. Atomicity (one CAS per multi-table write, no observable partial state), isolation (one immutable snapshot per operation, typed read-set conflicts) and consistency (validation before any effect) are unchanged. The one thing deliberately not offered is durability of effort: a prepared-but-uncommitted attempt is not a promise to anyone, and nothing hands out a prepare acknowledgement.
The only pre-publication file any writer stages is the schema contract (schema apply and the system-column upgrade), and its marker binds it to the publishing commit: lineage decides install or discard — there is no plan to interpret.
Merge with main
The stack predates #691/#742; this PR merges
mainand reconciles: the recovery probe (ensure_no_pending_recovery) is reimplemented over the legacy-sidecar refusal with identical messages and bounds; the engine tests are ported to the session API (helpers::session,with_setting("ann_nprobes", …)instead of engine-read env vars); the RFC-numbered namespace closure is extended to 0067/0068, which were reserved by open PR #725 when it closed; the four GQT store-place cases whose subject was the mutation's sidecar writes are retired (the store-place machinery itself lands unchanged); and the V5↔V9 fence naming follows to V10.Validation (local, on this exact tree)
Engine suite with failpoints (51 targets), DST from its crate directory with the cost golden regenerated byte-identical, GQT crate tests and full corpus, the remaining workspace crates with failpoints, both workspace clippy graphs plus GQT and DST clippy, fmt, docs/typos/AGENTS/action-pin checks, and the OpenAPI drift test. Earlier per-step validation (failure-window matrices per writer, full-mode 5-actor runs, S3 cells against RustFS) ran on each stacked PR, all CI-green before consolidation.
Coverage completion (a1136f4)
A first-principles audit of the protocol's test surface closed the three gaps that mattered:
__manifestcommit puts, the publication door itself. Under seeded schedules the presumed-abort semantics absorb every schedule (31 acks lost, 11 client retries converging against their own durable-but-denied commits in the pinned universe), with no model changes needed — which is itself the strongest evidence the protocol's crash story is complete.schema_apply.*crash windows are exercised (the post-publish window proven absorbing — promotion is advisory), and the revived persisted-write lie verbs found the engine answering a misdirected__schema_state.jsoninstall with a loud typed refusal, pinned as a detected-or-harmless canary. First seed scan also surfaced the engine's mono-branch apply restriction, now modeled as a predicted legal refusal.Load(loader staging over the mutation seams),FtsRebuild(segment replacement over the ensure-indices seams), andSystemColumnUpgrade(RFC 0040 on a graph born legacy, over the schema-apply seams, with a vintage-check oracle and sentinel-refused racers — its first process-kill coverage). The matrix is now ten writers, 246 cells, ~39s.All pinned DST plans keep their exact draw sequences (zero-knob discipline); the full DST suite, both clippy graphs, the matrix, and the repo checks are green on the exact tree.