schema-lint chassis v1.2: --allow-data-loss flag + Hard mode (MR-694) — completes v1

[aaltshuler]

Summary

Final chassis v1 commit. Wires up the --allow-data-loss CLI flag and Hard mode for both DropProperty and DropType. After this lands, the chassis v1 series (MR-694) is complete and the chassis epic + MR-700 (DropType/DropProperty) ticket can close.

What lands

CLI (omnigraph-cli):

• New --allow-data-loss flag on both omnigraph schema plan and omnigraph schema apply. Off by default → Soft drops (today's behavior).
• HTTP remote schema apply explicitly rejects the flag for now with a clear error. HTTP parity is a separate small follow-up.

Engine (omnigraph + omnigraph::db):

• New pub struct SchemaApplyOptions { pub allow_data_loss: bool }, re-exported as omnigraph::db::SchemaApplyOptions.
• New public methods plan_schema_with_options / apply_schema_with_options. Existing plan_schema / apply_schema are thin wrappers using Default::default(). No breaking changes for existing callers.
• promote_drops_to_hard: post-plan walk that promotes every DropMode::Soft to DropMode::Hard when the flag is set. Keeps the compiler's plan_schema_migration signature unchanged — promotion lives in the engine layer where operator intent belongs.
• New hard_cleanup_targets: Vec<(String, String)> accumulator. Apply path for both DropProperty and DropType populates it when the variant is Hard.
• Post-publish cleanup: new loop iterates hard_cleanup_targets and calls cleanup_old_versions (before_timestamp = now) on each dataset URI. Best-effort — the apply is already durable; cleanup failure is tracing::warn, not a hard failure.

Tests (tests/schema_apply.rs):

• apply_schema_with_allow_data_loss_promotes_drops_to_hard — default plan emits Soft; with options.allow_data_loss=true, plan emits Hard; apply succeeds. Negative: no Soft drops remain in the promoted plan.
• apply_schema_hard_drops_property_makes_prior_version_unreachable — the key Hard-mode behavioral test. After Hard drop + cleanup, snapshot_at_version(pre_drop).open("node:Person") FAILS because Lance reclaimed the prior dataset version. This is the user-facing Hard contract.
• apply_schema_hard_drops_node_and_edge_with_flag_succeeds — both Node and Edge DropType variants promote to Hard with the flag; apply succeeds; current manifest entries gone.

Behavioral details

DropProperty Hard: stage_overwrite produced a new dataset version without the column. cleanup_old_versions removes the prior version + reclaims unique fragments. Post-apply, time-travel back to pre-drop fails (Lance prior version is gone).

DropType Hard: no per-table write happens — the change is the manifest tombstone. cleanup_old_versions on the orphan dataset is a no-op in the immediate term (no prior versions to clean since the dataset wasn't modified by this apply). The dataset directory persists. User-facing contract: data is unreachable via omnigraph (manifest entry tombstoned). Full orphan-cleanup is a documented follow-up.

Test plan

• cargo test -p omnigraph-compiler --lib — 239 pass
• cargo test -p omnigraph-engine --test schema_apply — 14 pass (3 new + 11 existing)
• cargo test -p omnigraph-server --test openapi — 60 pass (no HTTP changes; OpenAPI unchanged)

What's deliberately not in this PR (known follow-ups)

• HTTP parity — extend SchemaApplyRequest with allow_data_loss: bool field, thread through server handler, regenerate openapi.json. Small focused PR.
• Orphan-dataset directory deletion for DropType { Hard } — currently the dataset directory persists after Hard type drop because cleanup_old_versions doesn't remove it (no prior versions to clean). Needs object_store recursive delete. Separate ticket.
• MR-948 substrate alignment — swap DropProperty Soft from stage_overwrite (O(rows)) to Lance native Dataset::drop_columns (O(catalog)). Cost-class optimization; functional behavior unchanged.

v1 status

After this lands, chassis v1 is complete for CLI/embedded use:

Commit	What	Where
#1+2	chassis v0 + DropMode dormant	merged in #90
#3	DropProperty { Soft } end-to-end	merged in #90
#4	DropType { Soft } end-to-end	merged in #99
#5	--allow-data-loss flag + Hard mode	this PR

MR-694 chassis epic + MR-700 DropType/DropProperty ticket can close after merge.

🤖 Generated with Claude Code

---

[devin-ai-integration]

Devin Review found 1 potential issue.

View 4 additional findings in Devin Review.

[devin-ai-integration]

🔴 AGENTS.md Rule 1 violation: new --allow-data-loss CLI flag ships without user-facing doc updates

AGENTS.md Rule 1 mandates: "New endpoint, query function, CLI flag, env var, constant, schema construct, or invariant: update both the source code and the doc in the same change. Never split documentation drift into a follow-up." This PR adds --allow-data-loss to schema plan and schema apply but none of the user-facing docs are updated: docs/user/cli-reference.md doesn't mention the flag, docs/user/schema-language.md doesn't document DropProperty/DropType steps or the soft/hard mode distinction, and docs/user/maintenance.md doesn't describe the hard-drop cleanup behavior. The SchemaApplyOptions public API type and the plan_schema_with_options/apply_schema_with_options public methods are also undocumented.

Prompt for agents

AGENTS.md Rule 1 requires that new CLI flags, API methods, and behavioral changes are documented in the same PR. This PR adds --allow-data-loss to schema plan and schema apply commands but does not update any user-facing docs.

Files to update:
1. docs/user/cli-reference.md — add --allow-data-loss to the schema plan and schema apply entries, explaining that it promotes soft drops to hard drops (cleanup_old_versions runs post-apply).
2. docs/user/schema-language.md — add DropProperty and DropType to the migration step type list (they were added in a prior PR but never documented). Document the Soft vs Hard mode distinction and the --allow-data-loss flag.
3. docs/user/maintenance.md — mention that hard-drop schema apply runs cleanup_old_versions inline on affected datasets, and note the DropType Hard limitation (dataset directory persists until orphan-cleanup pass).

Also consider updating the quick-reference flows in AGENTS.md to show --allow-data-loss usage.

---

Was this helpful? React with 👍 or 👎 to provide feedback.

[cubic-dev-ai]

cubic analysis

1 issue found across 5 files

Prompt for AI agents (unresolved issues)

Check if these issues are valid — if so, understand the root cause of each and fix them. If appropriate, use sub-agents to investigate and fix each issue separately.


<file name="crates/omnigraph/src/db/omnigraph/schema_apply.rs">

<violation number="1" location="crates/omnigraph/src/db/omnigraph/schema_apply.rs:214">
P1: Hard DropProperty resolves cleanup target from the renamed key, which can fail valid rename+drop migrations with `--allow-data-loss`.</violation>
</file>

Linked issue analysis

Linked issue: MR-694: Schema-lint chassis: classification tuple + per-rule severity + suppression + pre-migration checks

Status	Acceptance criteria	Notes
✅	Add --allow-data-loss flag to `omnigraph schema plan` and `omnigraph schema apply`; remote (HTTP) apply explicitly rejects the flag	CLI argv parsers in crates/omnigraph-cli/src/main.rs add `allow_data_loss` for both plan and apply and return an error when a remote HTTP apply is attempted with the flag.
✅	Introduce `SchemaApplyOptions` type and re-export it from `db` module; provide `plan_schema_with_options` / `apply_schema_with_options` while keeping `plan_schema` / `apply_schema` wrappers for compatibility	New struct `SchemaApplyOptions` added and re-exported; Omnigraph gains `plan_schema_with_options` and `apply_schema_with_options` and the old methods delegate to the new ones with defaults, preserving API shape.
✅	Planner promotes `DropMode::Soft` -> `DropMode::Hard` when `allow_data_loss` is set (plan-time promotion)	A `promote_drops_to_hard` function is defined and invoked during planning (and before apply) to mutate the plan when options.allow_data_loss is true; tests assert the promotion.
✅	Apply path records hard-drop cleanup targets for Hard-mode DropProperty and DropType and runs cleanup_old_versions after publish (best-effort, warn on failure)	The apply path builds `hard_cleanup_targets` for Hard drops and runs `cleanup_dataset_old_versions` after manifest publish; errors are logged as warnings rather than failing the apply.
✅	Hard-mode semantics enforced: DropProperty Hard causes prior dataset version to be reclaimed so time-travel pre-drop snapshot open fails	Tests exercise a Hard DropProperty apply and assert that opening the pre-drop snapshot fails (the code calls Lance cleanup_old_versions with before_timestamp=now to remove prior versions).
✅	Hard-mode DropType handling: tombstone manifest entry and run cleanup_old_versions on orphan dataset (data becomes unreachable via manifest), with note that directory deletion is a follow-up	Apply path tombstones the manifest entry and, for Hard, schedules cleanup_old_versions on the dataset; PR documents that directory deletion is a separate follow-up. Tests assert manifest entries gone and that apply succeeds.
✅	Tests added/updated to cover promotion, apply success with flag, and Hard-mode behavior; CI test targets indicated as passing	New tests in crates/omnigraph/tests/schema_apply.rs cover plan promotion, apply with allow_data_loss, and the Hard-mode contracts. PR test-plan summary lists test runs and passing counts; code changes include the tests themselves.

_{Reply with feedback, questions, or to request a fix. Tag @cubic-dev-ai to re-run a review.
Re-trigger cubic}

[cubic-dev-ai]

P1: Hard DropProperty resolves cleanup target from the renamed key, which can fail valid rename+drop migrations with --allow-data-loss.

Prompt for AI agents

Check if this issue is valid — if so, understand the root cause and fix it. At crates/omnigraph/src/db/omnigraph/schema_apply.rs, line 214:

<comment>Hard DropProperty resolves cleanup target from the renamed key, which can fail valid rename+drop migrations with `--allow-data-loss`.</comment>

<file context>
@@ -145,51 +191,71 @@ pub(super) async fn apply_schema_with_lock(
                     changed_edge_tables = true;
                 }
+                if matches!(mode, DropMode::Hard) {
+                    let entry = snapshot.entry(&table_key).ok_or_else(|| {
+                        OmniError::manifest(format!(
+                            "missing table '{}' for hard property drop",
</file context>

Suggested change

	let entry = snapshot.entry(&table_key).ok_or_else(|| {
	OmniError::manifest(format!(
	"missing table '{}' for hard property drop",
	table_key
	))
	})?;
	let full_uri = format!("{}/{}", db.root_uri, entry.table_path);
	hard_cleanup_targets.push((table_key.clone(), full_uri));
	let snapshot_key = renamed_tables.get(&table_key).unwrap_or(&table_key);
	let entry = snapshot.entry(snapshot_key).ok_or_else(|| {
	OmniError::manifest(format!(
	"missing table '{}' for hard property drop",
	snapshot_key
	))
	})?;
	let full_uri = db.table_store.dataset_uri(&entry.table_path);
	hard_cleanup_targets.push((table_key.clone(), full_uri));