docs(cluster): RFC-004 — graph & schema apply design (Phase 4)

[aaltshuler]

The design slice the implementation spec's exit criteria require before any graph-moving cluster apply code ships. Docs-only.

What the RFC settles

1. Roll-forward-only cluster recovery (the load-bearing simplification): the engine's own recovery sidecars already make every graph-level operation atomic within the graph, so the cluster never rolls a graph back — its sidecars (__cluster/recoveries/{ulid}.json, schema in §D2) exist to classify and record, converging the JSON ledger to observable reality (axiom 5) or surfacing a loud actual_applied_state_pending condition. An eight-row decision matrix (§D3) covers every crash window, each row testable with the Stage 3B failpoint harness; the sweep's only mutations are ordinary CAS-checked state writes under the existing lock.
2. Digest-bound approval artifacts (§D4) for the irreversible tier (graph delete, allow_data_loss schema apply): written by a new cluster approve command, consumed by apply only on exact digest match, retired into state.approval_records (axiom 11's durable audit reference). allow_data_loss is never a CLI flag on apply.
3. Actor threading (§D5): cluster apply --actor flows to the engine's apply_schema_as, so Cedar enforcement and commit attribution work unchanged; the cluster adds no policy engine.
4. Ordering and groups (§D5): creates → schema applies → catalog → deletes, per-resource groups, cross-graph atomicity explicitly not promised; the single final state CAS of Stage 3A is preserved.
5. Staging (Sequencing): 4A graph create (moves no existing manifest) → 4B schema apply (the heart) → 4C graph delete (irreversible tier last), each a separate PR gated on per-matrix-row failpoint tests.

Dispositions the implementation spec's high-risk decisions: #2 (derived graph roots), #3 (nested apply under sidecar coverage), #7 (the recovery sweep never auto-deletes). Exit-criteria coverage table maps 2 and 4 answered fully, 1/5/6 partially, 3/7/8/9 deferred to their phases.

Also: dev-index row under Active Implementation Plans, and a pointer from the implementation spec's Phase 4 section.

Verification

• scripts/check-agents-md.sh — green (new doc linked from docs/dev/index.md)
• Recovery matrix cross-checked against the engine classifier semantics in db/manifest/recovery.rs
• Axioms walk noted inline (§D8): 5, 8, 9, 11, 12, 15 + the deny-list's no-custom-transaction-manager and loud-partials rules

🤖 Generated with Claude Code

Greptile Summary

This docs-only PR introduces RFC-004, the Phase 4 design for cluster graph and schema apply. It establishes the cluster recovery sidecar schema, a roll-forward-only decision matrix for crash recovery, digest-bound approval artifacts for irreversible operations, actor threading through to the engine's apply_schema_as, and a three-stage sequencing plan (4A graph create → 4B schema apply → 4C graph delete).

• D3 recovery matrix: The eight-row (plus 7b) matrix is the load-bearing artefact of the RFC; it has two normative gaps — row 1's "any" kind can incorrectly match graph_delete sidecars, and the "only rows 3/4/7b mutate state" claim conflicts with rows 5/6/8 also setting persistent status/condition fields.
• D4 approval artifacts and D5 actor/ordering: The design is self-consistent, correctly defers allow_data_loss to an explicit approval path, and preserves the single-final-CAS invariant from Stage 3A.
• D6–D8: Graph delete orchestration, plan/import integration, and the invariant/axiom walk are all internally consistent and correctly tied back to the deny-list and existing engine behavior.

Confidence Score: 3/5

The RFC is a design document, but two normative gaps in the D3 decision matrix could lead to implementation bugs — one of which would silently suppress the graph_delete_incomplete diagnostic for partial prefix-removal crashes, violating the loud-partials invariant the RFC explicitly commits to.

Both gaps are in the D3 recovery matrix, the load-bearing implementation contract. The row 1 / graph_delete overlap is a real ambiguity: a mid-removal crash leaves the manifest at its pre-deletion version, satisfying row 1's condition; a top-to-bottom sweep would delete the sidecar and emit no graph_delete_incomplete condition, contradicting the loud-partials invariant. The only-rows-3/4/7b-mutate claim needs qualification because rows 5/6/8 also require CAS-checked state writes to enforce the cross-command drifted/error blocking the RFC describes.

docs/dev/rfc-004-cluster-graph-schema-apply.md — specifically the D3 matrix (rows 1, 5, 6, 8 and the only-mutations claim).

Important Files Changed

Filename	Overview
docs/dev/rfc-004-cluster-graph-schema-apply.md	New 210-line RFC specifying the Phase 4 cluster apply design; two normative gaps in the D3 decision matrix: row 1's "any" kind can incorrectly match graph_delete sidecars (masking partial prefix-removal failures), and the claim that only rows 3/4/7b are state mutations is inconsistent with rows 5/6/8 requiring persistent status writes.
docs/dev/cluster-config-implementation-spec.md	Four-line additive pointer to the new RFC inserted before the existing Phase 4 bullet list; correct and consistent with the surrounding spec.
docs/dev/index.md	Single row added to the Active Implementation Plans table linking to the new RFC; description matches the RFC content and the link target exists.

Sequence Diagram

sequenceDiagram
    participant Op as Operator
    participant CLI as cluster apply
    participant Lock as State Lock
    participant Sidecar as __cluster/recoveries/
    participant Engine as Omnigraph Engine
    participant State as state.json (CAS)

    Op->>CLI: cluster apply
    CLI->>Lock: acquire
    CLI->>CLI: run sweep (D3 matrix)
    CLI->>Sidecar: write sidecar (observed_manifest_version)
    CLI->>Lock: release

    CLI->>Engine: graph_create / schema_apply / graph_delete
    Engine-->>CLI: SchemaApplyResult (manifest_version)

    CLI->>Sidecar: rewrite sidecar (expected_manifest_version)

    CLI->>Lock: acquire
    CLI->>State: CAS write (all outcomes + approval_records)
    CLI->>Sidecar: delete sidecar
    CLI->>Lock: release

    Note over CLI,State: Crash anywhere leaves sidecar on disk
    Note over CLI,State: Next state-mutating command runs sweep (rows 1-8)

Loading

_{Reviews (2): Last reviewed commit: "docs(cluster): RFC-004 review fixes — gr..." | Re-trigger Greptile}