engine: schema apply stages detached rewrites and publishes its contract without a sidecar (RFC 0067 step 4)

Schema apply is the fourth writer moved to detached table commits.

An existing-table rewrite is a detached Overwrite of the promoted HEAD,
published as a pin one past the published version and promoted after the
manifest commit. An added type stays a linear version-one create at its
identity path; that path is a deterministic function of the accepted
identity allocator (the RFC's "fresh incarnation path" was wrong), so an
attempt that died after creating the dataset left it exactly where the
retry creates it, and the retry reclaims the unregistered leftover under
the schema sentinel instead of refusing to claim unowned physical state.

The schema contract is staged with the graph commit that publishes it
recorded in `__schema_state.json.staging`, and the writer installs the
live contract from memory after the commit. No SchemaApply sidecar is
written: a failure before the manifest commit returns the plain error
and leaves detached versions, a created dataset and a staged contract
that the next read-write open discards; a failure after it reports
`RecoveryRequired` naming the published commit, and the next read-write
open, or the same handle's next write, installs the contract because
that commit is in main's lineage. A read-only open refuses a published
but uninstalled contract and serves an unpublished staging as absent.
The open also reclaims a sentinel left by a crashed apply, which
previously stayed behind whenever an apply died after deleting its
sidecar, and the release is idempotent.

The RFC 0040 system-column upgrade keeps the v9 exact protocol, its
unmarked staging and its sidecar; the schema-apply sidecar constructor
is test-only until step 5 removes the classifier.

Lance fence: an Overwrite replayed over its own twin is idempotent
(Lance recognises the committed transaction and adds no version), so
racing promoters of a schema-apply pin leave no residue.

Tests: the schema-apply failpoint cells become no-residue, staging
discard, staging promotion, leftover reclaim, same-handle heal and
manifest-CAS-loss tests; the sidecar-rollback, foreign-winner and
Optimize late-sidecar tests whose producer no longer exists are
retired; `detached_commit_matrix` gains the `SchemaApply` writer (52
cells in the full run); `write_cost` gains
`schema_apply_writes_no_control_object`; the seam
`schema_apply.post_publish_pre_promotion` is new and the DST catalog
lists 72 windows.

Author: ragnorc

crates/omnigraph-dst/cost_table.txt

@@ -65,12 +65,12 @@ UpdateV                      a.list       calls=4
 UpdateV                      l.get        calls=10
 UpdateV                      l.list       calls=20
 _audit                       a.delete     calls=1
-_audit                       a.exists     calls=79
+_audit                       a.exists     calls=81
 _audit                       a.get        calls=122
 _audit                       a.list       calls=4
 _audit                       a.put_if_absent calls=1
-_audit                       l.get        calls=1228
-_audit                       l.list       calls=65
+_audit                       l.get        calls=1230
+_audit                       l.list       calls=66
 _close                       a.exists     calls=6
 _close                       a.get        calls=10
 _close                       a.list       calls=2
@@ -87,9 +87,9 @@ _setup                       a.list       calls=3
 _setup                       a.put        calls=2
 _setup                       a.put_if_absent calls=4
 _setup                       l.get        calls=139
-_setup                       l.list       calls=37
+_setup                       l.list       calls=38
 _setup                       l.put        calls=35
-_verify                      a.exists     calls=376
+_verify                      a.exists     calls=389
 _verify                      a.get        calls=601
 _verify                      a.list       calls=13
 _verify                      l.get        calls=2069

crates/omnigraph-dst/src/catalog.rs

@@ -1,13 +1,13 @@
 //! The full crash-window catalog for the hunt
-//! (`dst_hunt_crash_window_sweep`): 71 of the engine's decision seams
+//! (`dst_hunt_crash_window_sweep`): 72 of the engine's decision seams
 //! (`omnigraph::seams::catalog`) at the pinned engine version. A seam added
 //! to the engine enters here as never-reached until its workload exists.
 //!
 //! Kept honest by `catalog_names_are_engine_seams` below: every entry must
 //! be a name the engine catalog declares, so a typo'd or renamed-away window
 //! fails the suite instead of compiling and silently never firing.
 
-pub const CRASH_WINDOWS: [&str; 71] = [
+pub const CRASH_WINDOWS: [&str; 72] = [
     "blob_read.post_capture",
     "branch_control.post_recovery_barrier",
     "branch_create.post_native",
@@ -74,6 +74,7 @@ pub const CRASH_WINDOWS: [&str; 71] = [
     "schema_apply.after_staging_write",
     "schema_apply.before_staging_write",
     "schema_apply.post_sidecar_pre_effect",
+    "schema_apply.post_publish_pre_promotion",
     "schema_apply.post_table_commit",
     "schema_reload.before_contract_read",
     "storage.local_create_if_absent_probe",

crates/omnigraph-dst/tests/scenarios.rs

@@ -2214,6 +2214,11 @@ fn dst_milestone_never_remerges_merged_branch() {
 /// The detached index writer opens each productive table at its pin and
 /// promotes the twin it lands (EnsureIndices l.get 11 -> 15; the closing
 /// pass reads the promoted twins, _close l.get 31 -> 35, _verify 2071 -> 2069).
+/// Detached schema apply: a read-write open lists the branch refs once to
+/// reclaim a stale schema-apply sentinel (_setup/_audit l.list 37/65 ->
+/// 38/66, _audit l.get 1228 -> 1230) and a read-only open probes the staged
+/// schema state once for its coherence proof (_audit a.exists 79 -> 81,
+/// _verify 376 -> 389).
 #[test]
 #[serial]
 fn dst_bench_cost_count_golden() {

crates/omnigraph/src/db/manifest.rs

@@ -70,10 +70,9 @@ pub(crate) use recovery::{
     HealPendingOutcome, MAX_BRANCH_MERGE_DATA_TRANSACTIONS, RecoveryAuthorityToken,
     RecoveryLineageIntent, RecoveryManifestDelta, RecoveryMode, RecoverySchemaApplyEffect,
     RecoverySchemaApplyEffectKind, RecoverySidecarHandle, RecoverySystemColumnUpgrade,
-    RecoveryTableUpdateSlot, SidecarKind, SidecarTablePin, SidecarTableRegistration,
-    SidecarTableRename, SidecarTombstone, confirm_schema_apply_sidecar_v9, delete_sidecar,
-    ensure_read_only_schema_coherent, heal_pending_sidecars_roll_forward, list_sidecars,
-    new_optimize_sidecar_v9, new_schema_apply_sidecar_v9, new_system_column_upgrade_sidecar_v9,
+    RecoveryTableUpdateSlot, SidecarKind, SidecarTablePin, confirm_schema_apply_sidecar_v9,
+    delete_sidecar, ensure_read_only_schema_coherent, heal_pending_sidecars_roll_forward,
+    list_sidecars, new_optimize_sidecar_v9, new_system_column_upgrade_sidecar_v9,
     recover_manifest_drift, schema_apply_serial_queue_key, write_sidecar,
 };
 pub use state::DatasetEntry;

crates/omnigraph/src/db/manifest/recovery.rs

@@ -43,8 +43,8 @@ use crate::branch_control::list_branch_contents;
 use crate::db::graph_coordinator::GraphCoordinator;
 use crate::db::recovery_audit::{RecoveryAudit, RecoveryAuditRecord, RecoveryKind, TableOutcome};
 use crate::db::schema_state::{
-    SchemaStateRecovery, read_schema_state_identity, schema_ir_staging_uri,
-    schema_source_staging_uri, schema_state_staging_uri,
+    read_schema_state_identity, schema_ir_staging_uri, schema_source_staging_uri,
+    schema_state_staging_uri,
 };
 use crate::error::{OmniError, Result};
 use crate::storage::StorageAdapter;
@@ -3265,7 +3265,6 @@ pub(crate) async fn heal_pending_sidecars_roll_forward(
             .iter()
             .map(|pin| (pin.table_key.clone(), pin.table_branch.clone()))
             .collect();
-        let is_schema_apply = matches!(sidecar.writer_kind, SidecarKind::SchemaApply);
         let _table_guards = write_queue.acquire_many(&queue_keys).await;
         // Re-read after the wait: the writer we blocked on may have completed
         // Phase C and deleted the sidecar, or may have durably confirmed Phase B
@@ -3283,21 +3282,6 @@ pub(crate) async fn heal_pending_sidecars_roll_forward(
         // It also re-runs per sidecar, so a multi-sidecar pass never
         // classifies against a reconcile result an earlier roll-forward
         // staled. Non-SchemaApply sidecars never consult the value.
-        let schema_state_recovery = if is_schema_apply {
-            let snapshot = {
-                let mut coord = coordinator.write().await;
-                coord.refresh().await?;
-                coord.snapshot()
-            };
-            crate::db::schema_state::recover_schema_state_files(
-                root_uri,
-                std::sync::Arc::clone(&storage),
-                &snapshot,
-            )
-            .await?
-        } else {
-            SchemaStateRecovery::Noop
-        };
         // Fresh per-branch snapshot — same rationale as
         // `recover_manifest_drift`: classify against the branch the
         // sidecar's writer targeted, refreshed after any prior
@@ -3336,7 +3320,6 @@ pub(crate) async fn heal_pending_sidecars_roll_forward(
             &branch_snapshot,
             &sidecar,
             RecoveryMode::RollForwardOnly,
-            schema_state_recovery,
         )
         .await?
         {
@@ -3484,7 +3467,6 @@ pub(crate) async fn recover_manifest_drift(
     storage: std::sync::Arc<dyn StorageAdapter>,
     coordinator: &mut GraphCoordinator,
     mode: RecoveryMode,
-    schema_state_recovery: SchemaStateRecovery,
     write_queue: &crate::db::write_queue::WriteQueueManager,
 ) -> Result<()> {
     let sidecars = list_sidecars(root_uri, storage.as_ref()).await?;
@@ -3554,15 +3536,7 @@ pub(crate) async fn recover_manifest_drift(
                 coordinator.snapshot()
             }
         };
-        process_sidecar(
-            root_uri,
-            &storage,
-            &branch_snapshot,
-            &sidecar,
-            mode,
-            schema_state_recovery,
-        )
-        .await?;
+        process_sidecar(root_uri, &storage, &branch_snapshot, &sidecar, mode).await?;
     }
     // Final refresh so the caller sees the post-sweep state.
     coordinator.refresh().await?;
@@ -3783,7 +3757,6 @@ async fn process_sidecar(
     snapshot: &Snapshot,
     sidecar: &RecoverySidecar,
     mode: RecoveryMode,
-    schema_state_recovery: SchemaStateRecovery,
 ) -> Result<bool> {
     // Returns whether durable state changed (roll-forward, roll-back,
     // effect-free retirement, or stale-sidecar audit recovery). `false` =
@@ -4188,9 +4161,7 @@ async fn process_sidecar(
                 .map(|()| true)
         }
         SidecarDecision::RollForward => {
-            if matches!(sidecar.writer_kind, SidecarKind::SchemaApply)
-                && !schema_state_recovery.completed_schema_apply_sidecar_rename()
-            {
+            if matches!(sidecar.writer_kind, SidecarKind::SchemaApply) {
                 if sidecar.schema_version == SCHEMA_APPLY_CONFIRMATION_SCHEMA_VERSION {
                     let target_is_live =
                         schema_apply_target_identity_is_live(root_uri, storage.as_ref(), sidecar)
@@ -5613,6 +5584,7 @@ async fn regenerate_system_column_upgrade_staging(
         root_uri,
         storage.as_ref(),
         &target.desired_ir,
+        None,
     )
     .await?;
     crate::db::schema_state::validate_exact_schema_staging_target(
@@ -6939,6 +6911,28 @@ pub(crate) async fn ensure_read_only_schema_coherent(
         }
     }
 
+    // RFC 0067: schema apply arms no sidecar. Its staged contract names the
+    // publishing graph commit; once that commit is in lineage the manifest
+    // already carries the new registrations, and only a read-write open may
+    // install the contract files. An unpublished staging is inert garbage.
+    if let crate::db::schema_state::StagedContract::Marked {
+        state,
+        published: true,
+    } = crate::db::schema_state::inspect_staged_contract(root_uri, storage, false).await?
+    {
+        let graph_commit_id = state
+            .publication
+            .map(|publication| publication.graph_commit_id)
+            .unwrap_or_default();
+        return Err(OmniError::recovery_required(
+            graph_commit_id.clone(),
+            format!(
+                "read-only open found SchemaApply manifest outcome for graph commit '{}' but the schema contract promotion is pending; run a read-write open to finish it",
+                graph_commit_id
+            ),
+        ));
+    }
+
     for sidecar in sidecars {
         if !matches!(sidecar.writer_kind, SidecarKind::SchemaApply) {
             continue;
@@ -8687,6 +8681,7 @@ pub(crate) async fn confirm_occ_sidecar_v9(
 /// independently durable table operation; metadata/tombstone-only applies may
 /// deliberately carry an empty set because schema staging is confirmed by the
 /// same protocol before publication.
+#[cfg(test)]
 pub(crate) fn new_schema_apply_sidecar_v9(
     actor_id: Option<String>,
     tables: Vec<SidecarTablePin>,

crates/omnigraph/src/db/omnigraph.rs

@@ -62,12 +62,13 @@ use super::manifest::{
     GenesisManifestAttempt, ManifestChange, Snapshot, TableRegistration, TableTombstone,
 };
 use super::schema_state::{
-    SCHEMA_SOURCE_FILENAME, SchemaContractText, load_validated_schema_contract,
-    load_validated_schema_contract_for_source, read_accepted_schema_ir, read_schema_contract_text,
-    read_schema_contract_text_for_source, read_schema_state_identity, recover_schema_state_files,
-    render_schema_contract, schema_ir_uri, schema_source_staging_uri, schema_source_uri,
-    schema_state_uri, validate_schema_contract, validate_schema_contract_text,
-    validate_schema_ir_against_snapshot, write_schema_contract, write_schema_contract_staging,
+    SCHEMA_SOURCE_FILENAME, SchemaContractText, SchemaStagingPolicy, SchemaStateRecovery,
+    load_validated_schema_contract, load_validated_schema_contract_for_source,
+    read_accepted_schema_ir, read_schema_contract_text, read_schema_contract_text_for_source,
+    read_schema_state_identity, recover_schema_state_files, render_schema_contract, schema_ir_uri,
+    schema_source_staging_uri, schema_source_uri, schema_state_uri, validate_schema_contract,
+    validate_schema_contract_text, validate_schema_ir_against_snapshot, write_schema_contract,
+    write_schema_contract_staging,
 };
 use super::{
     ReadTarget, ResolvedTarget, SCHEMA_APPLY_LOCK_BRANCH, SnapshotId, is_internal_system_branch,
@@ -213,6 +214,11 @@ pub struct Omnigraph {
     coordinator: Arc<tokio::sync::RwLock<GraphCoordinator>>,
     table_store: TableStore,
     runtime_cache: RuntimeCache,
+    /// RFC 0067: this handle's schema apply published its manifest commit but
+    /// could not install the schema contract. The next write-entry heal on
+    /// this handle installs it from the staged copy; other handles and
+    /// processes converge at their next read-write open.
+    pending_schema_install: std::sync::atomic::AtomicBool,
     /// Warm change-feed cut for this handle's bound branch. A cut (head,
     /// witness, genesis, lineage projection, forward child index) is a PURE
     /// projection of `__manifest`, so it is exactly valid while the manifest
@@ -495,7 +501,7 @@ impl Omnigraph {
         let schema_identity_domain = schema_ir.schema_identity_domain.as_str().to_string();
         let mut catalog = build_catalog_from_ir(&schema_ir)?;
         fixup_physical_schemas(&mut catalog)?;
-        let (_, ir_json, state_json) = render_schema_contract(&schema_ir)?;
+        let (_, ir_json, state_json) = render_schema_contract(&schema_ir, None)?;
         let contract = SchemaContractText {
             source: schema_source.to_string(),
             ir_json,
@@ -665,6 +671,7 @@ impl Omnigraph {
             // sessions reuse the process-wide object-store registry.
             table_store: TableStore::new(&root, session),
             runtime_cache: RuntimeCache::default(),
+            pending_schema_install: std::sync::atomic::AtomicBool::new(false),
             feed_cut_cache: tokio::sync::RwLock::new(None),
             read_caches,
             schema_view: Arc::new(ArcSwap::from_pointee(HandleSchemaView {
@@ -784,12 +791,29 @@ impl Omnigraph {
         if matches!(mode, OpenMode::ReadWrite) {
             // Schema staging is itself mutable recovery state. Hold the shared
             // schema gate across BOTH its file pre-pass and the complete Full
-            // sidecar sweep, so `schema_state_recovery` cannot go stale in a
+            // sidecar sweep, so the schema staging decision cannot go stale in a
             // release/reacquire gap. The sweep adds branch → sorted table gates
             // per sidecar under this outer guard.
-            let schema_state_recovery =
-                recover_schema_state_files(&root, Arc::clone(&storage), &coordinator.snapshot())
-                    .await?;
+            recover_schema_state_files(
+                &root,
+                Arc::clone(&storage),
+                &coordinator.snapshot(),
+                SchemaStagingPolicy::PromoteOrDiscard,
+            )
+            .await?;
+            // RFC 0067: a crashed schema apply leaves its durable sentinel
+            // behind with no sidecar to retire it. The open-time pass above
+            // settled its staging, so the sentinel is stale under the same
+            // one-mutation-process boundary; reclaim it before the sweep.
+            if coordinator
+                .all_branches()
+                .await?
+                .iter()
+                .any(|branch| is_schema_apply_lock_branch(branch))
+            {
+                tracing::warn!("reclaiming the schema apply sentinel left by a crashed apply");
+                coordinator.branch_delete(SCHEMA_APPLY_LOCK_BRANCH).await?;
+            }
             // Recovery sweep: close the Phase B → Phase C residual on
             // any sidecar left over from a crashed writer. Long-running
             // processes additionally converge in-process: the staged-
@@ -802,7 +826,6 @@ impl Omnigraph {
                 Arc::clone(&storage),
                 &mut coordinator,
                 crate::db::manifest::RecoveryMode::Full,
-                schema_state_recovery,
                 write_queue.as_ref(),
             )
             .await?;
@@ -857,6 +880,7 @@ impl Omnigraph {
             // sessions reuse the process-wide object-store registry.
             table_store: TableStore::new(&root, session),
             runtime_cache: RuntimeCache::default(),
+            pending_schema_install: std::sync::atomic::AtomicBool::new(false),
             feed_cut_cache: tokio::sync::RwLock::new(None),
             read_caches,
             schema_view: Arc::new(ArcSwap::from_pointee(HandleSchemaView {
@@ -2050,12 +2074,25 @@ impl Omnigraph {
             {
                 let mut coord = self.coordinator.write().await;
                 coord.refresh().await?;
-                recover_schema_state_files(
+                let outcome = recover_schema_state_files(
                     &self.root_uri,
                     Arc::clone(&self.storage),
                     &coord.snapshot(),
+                    SchemaStagingPolicy::PromoteOnly,
                 )
                 .await?;
+                // A promoted staging completes a published apply whose writer
+                // died before releasing its sentinel; release it here so the
+                // caller's write is not refused until the next open.
+                if matches!(outcome, SchemaStateRecovery::Promoted)
+                    && coord
+                        .all_branches()
+                        .await?
+                        .iter()
+                        .any(|branch| is_schema_apply_lock_branch(branch))
+                {
+                    coord.branch_delete(SCHEMA_APPLY_LOCK_BRANCH).await?;
+                }
             }
         } // ← guards released before the heal's queue acquisition
         let _outcome = crate::db::manifest::heal_pending_sidecars_roll_forward(
@@ -2330,12 +2367,38 @@ impl Omnigraph {
             &self.write_queue,
         )
         .await?;
-        if outcome.processed_any {
-            // A rolled-forward SchemaApply sidecar moved disk + manifest
-            // to the new schema (staging promoted, registrations
-            // published); the in-memory catalog must follow or the very
-            // write that triggered the heal validates against the stale
-            // schema. Same post-heal step as `refresh`.
+        // RFC 0067: finish this handle's own published-but-uninstalled schema
+        // apply before the caller's write plans against the manifest. The
+        // flag keeps the common path free of any staging probe.
+        let mut installed = false;
+        if self
+            .pending_schema_install
+            .swap(false, std::sync::atomic::Ordering::SeqCst)
+        {
+            let result = {
+                let _serial = self
+                    .write_queue
+                    .acquire(&crate::db::manifest::schema_apply_serial_queue_key())
+                    .await;
+                let snapshot = self.coordinator.read().await.snapshot();
+                recover_schema_state_files(
+                    &self.root_uri,
+                    Arc::clone(&self.storage),
+                    &snapshot,
+                    SchemaStagingPolicy::PromoteOnly,
+                )
+                .await
+            };
+            match result {
+                Ok(recovery) => installed = matches!(recovery, SchemaStateRecovery::Promoted),
+                Err(error) => {
+                    self.pending_schema_install
+                        .store(true, std::sync::atomic::Ordering::SeqCst);
+                    return Err(error);
+                }
+            }
+        }
+        if outcome.processed_any || installed {
             self.reload_schema_if_source_changed().await?;
             self.invalidate_read_caches().await;
         }