feat(engine): unify constraint validation across all write surfaces - #314
Conversation
Constraint enforcement (value/range/check, enum, uniqueness, edge referential integrity, cardinality) was implemented three times — once each in the bulk loader, the mutation executor, and the branch-merge path — and had drifted: merge validated @range/@check but not enum, and neither the mutation nor the load path enforced cross-version uniqueness against already-committed rows. Introduce one catalog-derived evaluator (`crate::validate`) that all three surfaces route through. It is delta-scoped (checks only the change set, not the whole graph) and index-backed (probes committed state through the @key/@unique/src/dst BTREEs instead of full-scanning every catalog table), reusing the existing leaf checks (validate_value_constraints, validate_enum_constraints, composite_unique_key) so the surfaces cannot drift again. A one-row-delta merge now opens ~3 data tables instead of ~6+, and validation cost is flat in graph size rather than O(V+E). Behavior changes (all stricter, none relaxed): - Enum constraints are now enforced on the merge path (was a gap). - A write or load whose @unique value collides with an already-committed different row is now rejected (cross-version uniqueness); re-upserting an existing @key still upserts. - Uniqueness distinguishes a duplicate key WITHIN one input batch (two distinct records -> rejected, e.g. a bulk load listing a @key twice) from the SAME id reappearing ACROSS batches (ordered supersession of one logical row -> coalesced, e.g. a mutation insert-then-update). - Overwrite loads validate per-table: a touched table's committed view is its replacement image (empty), but a table absent from the batch keeps its committed rows, so an edges-only overwrite still resolves referential integrity against retained nodes. Remove the per-surface validation orchestration the evaluator supersedes, and the now-orphaned version-pinned dataset opener from the sealed storage trait (reads route through the snapshot path). Docs (invariants, testing) updated; full engine suite green.
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 0ee6c41075
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| let projection: Vec<&str> = projection.iter().map(String::as_str).collect(); | ||
| let mut change = crate::validate::TableChange::default(); | ||
| match candidate { | ||
| CandidateTableState::AdoptSourceState => continue, |
There was a problem hiding this comment.
Include pointer-switch adopts in merge validation
When a source table is adopted by pointer switch/fork, it can still carry source-side row changes even though it does not advance Lance HEAD. For example, merging main into a target branch that deleted a node while main added an edge referencing that node classifies the edge table as AdoptSourceState (source on main, target branch pointer switch); skipping it here means EdgeRi never sees the new edge, while the target node deletion is kept, so the merge can publish an orphan edge. Represent this adopt as a source-vs-target delta for validation, or otherwise validate the adopted source table against the target snapshot before publishing.
Useful? React with 👍 / 👎.
| // BTREE (a non-indexed `@unique` column falls back to a scan). | ||
| let mut expr: Option<Expr> = None; | ||
| for (column, value) in columns.iter().zip(key.iter()) { | ||
| let eq = col(column.as_str()).eq(lit(value.clone())); |
There was a problem hiding this comment.
Preserve typed literals for unique probes
For @unique columns that are not strings, the key values here are already stringified by composite_unique_key, but lit(value.clone()) makes the committed-state lookup compare the real Arrow column (e.g. Date32, Date64, numeric, bool) to a Utf8 literal. In accepted schemas such as due: Date @unique or numeric composite uniques, the new cross-version check can miss the existing holder or fail with a coercion error even though intra-delta uniqueness supports those scalar types. Build the filter literal from the catalog/column type rather than the display key string.
Useful? React with 👍 / 👎.
| for (id, src) in &delta_edges { | ||
| per_src.entry(src.clone()).or_default().insert(id.clone()); |
There was a problem hiding this comment.
Deduplicate merge-load edge deltas before @card
LoadMode::Merge still commits edge batches with last-writer-wins dedupe by id, but this validator counts every raw delta edge. If a merge-load input contains the same edge id twice with different src values, the final table keeps only the last row while the @card check can count that id under multiple sources, producing spurious violations (the removed count_pending_src_with_dedupe handled this case). Normalize merge-load edge deltas by id before adding them to per_src.
Useful? React with 👍 / 👎.
Regression for a gap in the unified merge validation: when a table is adopted by pointer switch (AdoptSourceState) — source on main, target on a branch — build_merge_changeset skips it, so referential integrity is never checked for it. Merging main into a branch that deleted a node while main added an edge to that node silently publishes the orphan edge. This test merges main -> feature where feature deleted Bob and main added Knows Alice->Bob, and asserts an OrphanEdge conflict. Red against HEAD (merge returns Merged); turns green with the AdoptSourceState validation fix.
The unified merge validator skipped any table classified AdoptSourceState
(a pointer switch / fork), so referential integrity, uniqueness, and
cardinality were never checked for it. Merging main into a branch that
deleted a node while main added an edge to that node silently published the
orphan edge — the prior full-scan validation caught it.
Root cause: classify_adopt keyed AdoptSourceState on the publish mechanism
("does it advance Lance HEAD") and returned before computing any delta, and
build_merge_changeset then skipped the table. Fix decouples the validation
input from the publish mechanism: classify_adopt now always computes the
source-vs-target delta (base == target on this path, so it is the right
validation delta) and carries it as AdoptSourceState { validation_delta };
build_merge_changeset validates it exactly like AdoptWithDelta. The publish
stays a pointer/fork (delta ignored) and remains excluded from recovery
pins, so publish/recovery semantics are unchanged — only validation is
restored. Closes the class: no publish optimization can bypass validation.
Turns the orphan-edge regression test green.
The cross-version @unique check pushes a committed-state filter built from the stringified key. On a non-String @unique column (e.g. Date) this compares a Date32 column to a Utf8 literal — and the stringified key is the raw day count, so the probe raises "Cannot cast string '20633' to Date32" for ANY second write to the table (colliding or not). Two regressions: a colliding Date value must surface a proper "@unique violation" (not a coercion error), and a non-colliding write must succeed. Both red against HEAD; green with the typed-literal probe fix.
The cross-version @unique check pushed a Lance filter built with a stringified key (lit(String)) against the real, typed column. On a non-String @unique column this compared a Date32/numeric/bool column to a Utf8 literal: a coercion error on Date/Bool (failing every write to the table) or a silent miss on Float. For Date the stringified key was even the raw day count, so the literal could never parse. unique_holders now takes typed ScalarValues, built at the call site via ScalarValue::try_from_array(group_column, row), so the pushed-down predicate compares like-typed for any scalar @unique. The in-memory intra-delta dedup keeps the stringified key (a type-agnostic equality grouping, unaffected). Turns the Date @unique cross-version regression tests green.
Two cardinality drifts between validation and what commit persists: - Move (B): a Merge-load that moves an edge to a new src only recounts the new src, so vacating a src and dropping it below @card min is missed — moving Alice's only WorksAt to Bob silently succeeds under @card(1..). - Dup (A): a Merge-load batch listing one edge id under two srcs counts it under both, but commit dedupes by id (last-wins). Alice gets a phantom second edge and a spurious "has 2 edges (max 1)" violation under @card(0..1). Both red against HEAD; green with the id-keyed last-wins cardinality model.
@card validation diverged from what commit persists in two ways: (1) it only recounted the new src of a delta edge, so a Merge-load that moves an edge to a new src never rechecked the vacated src and missed a drop below @card min; (2) it counted raw delta rows, so the same edge id under two srcs in one batch was counted under both, while commit dedupes by id (last-wins) — a phantom edge and a spurious max violation. evaluate_cardinality now coalesces the delta by edge id (last-wins, matching dedupe_merge_batches_by_id) and builds the affected-src set from both the new src of each delta edge AND the old committed src of each changed/deleted edge id; a committed edge is dropped from its src when the delta deletes or re-places it. The validated edge set per src now equals the committed image. Turns the edge-move and duplicate-id cardinality regression tests green.
Proposed design for the by-design fix to merge cost/OOM: adopt the source branch's Lance fragments by reference (base_paths) instead of re-materializing rows, with a re-home reconciler + branch-delete reference guard closing the dangling-reference lifecycle, and a reachability-complete cleanup sweep. Grounded in the public Lance 7.0.0 multi-base APIs and the prior art (Delta shallow/deep clone, Iceberg/lakeFS reachability GC). Status: Proposed.
Deletes stage as predicates, not constructive batches, so a delete-only mutation produces an empty change-set and validate_changeset no-ops — a `delete WorksAt where from = X` that removes a source's only edge commits below @card(1..), while the merge path (which carries deleted_ids) rejects it. Red against HEAD (the delete commits); green once the delete path resolves its predicates into the validation change-set.
A delete-only mutation produced an empty change-set (deletes stage as predicates, not constructive batches), so validate_changeset no-op'd and a `delete Edge` that dropped a source below @card min committed silently — while the merge path, which carries deleted_ids, rejects it. validate_staged_mutation now resolves each staged delete predicate against the live committed table (CommittedState::deleted_ids_matching, a SQL-filter scan projecting id) and folds the matched ids into the change-set's deleted_ids for that table. The existing evaluator then recounts the srcs a delete empties (@card min) and sees removed rows for RI/node-delete — the same faithful change-set the merge path already builds, so validation matches what commits. Covers direct edge deletes, node deletes, and node-delete edge cascades uniformly (all are staged predicates). Turns the direct-edge-delete @card regression test green.
… re-scan The delete-cardinality fix resolved staged delete predicates a second time at validation. Instead, capture the removed ids during the delete op's own scan: execute_delete_edge and the node-delete edge cascade now scan id (not count_rows), record the ids via MutationStaging::record_deleted_ids, and to_changeset() folds them into the change-set's deleted_ids. validate_staged_ mutation reverts to plain to_changeset(); CommittedState::deleted_ids_matching and scan_filtered_sql are removed. Behavior-preserving (the @card-on-delete test stays green) and strictly fewer scans — one scan at delete time replaces count-here + resolve-at-validation. Node deletes already scanned their ids; this reuses that via a shared ids_from_batches helper. Full engine suite green; workspace builds clean.
There was a problem hiding this comment.
Cursor Bugbot has reviewed your changes and found 1 potential issue.
❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.
Reviewed by Cursor Bugbot for commit 9426e11. Configure here.
Two reviewer findings, both red against HEAD: - F1 (High): overwriting a node table removes nodes without expressing them as deleted_ids, so a retained edge in a non-overwritten table that references a removed node is published as an orphan (edge-RI path-b never runs). overwrite_node_removal_rejects_retained_orphan_edge. - F2 (Medium): evaluate_unique accumulates superseded keys across batches, so a mutation that frees a @unique value (Alice.email temp -> final) and reuses it (insert Carol.email = temp) false-rejects a valid final image. chained_unique_update_then_reuse_freed_value_is_not_a_violation.
An Overwrite load replaces each touched table, but to_changeset() only recorded the new batch, never the committed rows the overwrite removes. So overwriting node:Person to drop Bob while a retained edge:Knows(Alice->Bob) referenced him published an orphan edge unchecked — edge-RI path-b is gated on the node's deleted_ids, which were empty. The loader now computes per overwritten table the removed ids (committed ids in the pinned base minus the replacement batch's ids, via validate:: overwrite_removed_ids) and folds them into the change-set's deleted_ids. The evaluator then runs RI path-b and cardinality against them — the same faithful change-set the merge path builds. Overwrite is per-table, so a table absent from the batch is untouched; a removed node referenced by a retained edge is now a loud OrphanEdge. Updates two tests that asserted the old silent-orphan behavior to self-consistent overwrites (per-table Overwrite can't drop edge endpoints without also overwriting the edge tables): end_to_end::overwrite_replaces_data and writes::load_overwrite_with_bad_edge_reference_unblocks_next_load. The orphan-rejection case itself is pinned by the new validators test.
evaluate_unique iterated the raw delta batches and accumulated every key it saw into one cross-batch map, so a coalesced write that frees then reuses a @unique value within a query — update a row's email to 'temp', update the same row to 'final', insert a new row with 'temp' — false-rejected: 'temp' lingered in the seen-set from the superseded first write though it no longer holds in the final image that commits. Restructure to validate the final coalesced image — the bytes that actually publish: - Pass 1 coalesces the delta by id (last-wins) into each id's final key, and flags genuine within-ONE-batch duplicates (two distinct input records — the bulk-load contract) before coalescing, so an unordered load batch with a real dup still rejects. - Pass 2 checks two distinct final ids holding the same key. - Pass 3 does the committed cross-version lookup, excluding the delta's own ids. Entries are sorted by id before the cross-row/committed passes so violation order never depends on HashMap iteration. Coalescing first also drops the redundant committed probes a superseded key used to issue. Pinned by the chained-update red test; preserves intra-batch dup rejection (consistency::loader_rejects_intra_batch_duplicate_keys) and cross-version uniqueness (validators).
Left by a block-delete in an earlier refactor; flagged by git diff --check.
The module doc still said the merge path was the only consumer and the write path a later, mechanical migration, and listed cardinality as a later increment. Mutation and bulk load have since migrated onto the evaluator and cardinality ships — correct both so the doc reflects that all three write surfaces route through one evaluator.
…state Greptile P2: the prior wording implied a live-head @unique guarantee, but #314's check is snapshot-scoped (probes the write's pinned base view via the @key/@unique BTREEs), so a concurrent writer committing the same value after the base was opened is not caught — matching the 'full cross-version uniqueness is still a gap' note in docs/dev/invariants.md. Qualify with 'visible to that write'. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…linux-arm64 (#320) * docs(release): cover #314 stricter validation + #316 linux-arm64 in v0.8.0 notes Two changes landed after the version-bump commit (#313) that wrote the v0.8.0 release notes, so they were undocumented: - #314 unified constraint validation across the loader, mutation, and merge surfaces. Its behavior changes are all stricter (enum enforced on merge, cross-version @unique rejection, precise within-batch vs across-batch dup-key semantics, per-table overwrite validation) and are user-visible, so they get a dedicated section. - #316 added linux-arm64 (aarch64) as a first-class prebuilt target + Homebrew bottle; note the new platform. Also mention the stricter validation in the release intro. Docs-only. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * docs(release): scope cross-version uniqueness to the write's visible state Greptile P2: the prior wording implied a live-head @unique guarantee, but #314's check is snapshot-scoped (probes the write's pinned base view via the @key/@unique BTREEs), so a concurrent writer committing the same value after the base was opened is not caught — matching the 'full cross-version uniqueness is still a gap' note in docs/dev/invariants.md. Qualify with 'visible to that write'. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

What & why
Constraint enforcement (value/range/check, enum, uniqueness, edge referential integrity, cardinality) was implemented separately in the bulk loader, the mutation executor, and the branch-merge path, and had drifted — merge validated
@range/@checkbut not enum, and neither the mutation nor the load path enforced uniqueness against already-committed rows. This routes all three write surfaces through one catalog-derived, delta-scoped, index-backed evaluator (crate::validate) that reuses the existing leaf checks, closing the drift class by construction and making a small-delta merge's validation cost flat in graph size instead of O(V+E).Backing issue / RFC
Maintainer-internal change (part of the ongoing branch-ops / Lance-alignment work); per the template note above, the external accepted-issue/RFC link requirement does not apply to maintainers.
Checklist
validators.rscross-version uniqueness + per-table overwrite; newmerge_cost.rsdelta-scoped cost budget; equivalence viamerge_truth_table/branching)docs/dev/invariants.md,docs/dev/testing.md)Notes for reviewers
Behavior changes are all stricter, none relaxed:
@uniquevalue collides with an already-committed different row is now rejected (cross-version uniqueness); re-upserting an existing@keystill upserts.@keytwice) from the same id reappearing across batches (ordered supersession of one logical row → coalesced, e.g. a mutation insert-then-update).Overwriteloads validate per-table: a touched table's committed view is its replacement image (empty), but a table absent from the batch keeps its committed rows, so an edges-only overwrite still resolves referential integrity against retained nodes.Also removes the per-surface validation orchestration the evaluator supersedes (~600 lines) and the now-orphaned version-pinned dataset opener from the sealed storage trait (reads route through the snapshot path). Risk areas: the within-batch-vs-across-batch uniqueness split (relies on each mutation statement emitting its own batch and never repeating an id within one batch) and the index-backed committed probes (degrade to a tail scan over uncovered fragments — the same
optimizecadence dependency as the rest of the engine). Full engine suite green;cargo build --workspace --lockedclean.Note
Medium Risk
Touches all write paths and constraint semantics (stricter rejects); reviewers flagged a possible gap when an overwrite replaces an edge table without recounting
@cardmin for sources dropped from the new image.Overview
Introduces
crate::validate, a single catalog-derived evaluator that all three write surfaces use: merge, mutation, and bulk load. Per-op/per-table duplicate checks are removed in favor of end-of-operation validation over aChangeSet(added/changed batches plusdeleted_ids), with Δ-scoped, index-backed probes of committed state instead of full-graph scans.Merge drops full-table staging for validation and the inlined uniqueness/RI/cardinality loops; it builds a delta
ChangeSet(projecting out vector/blob columns) and routes throughevaluate.AdoptSourceStatenow carries an optionalvalidation_deltaso pointer/fork adopts are still checked when the source diverged. Mutations callvalidate_staged_mutationbefore commit; deletes record removed ids (including cascade) so@cardand RI see empties. Loader runs the same pass after staging, with per-tableOverwritemodeled viaoverwrite_removed_idsfor RI on retained edges.Behavior gets stricter: merge enforces enums; mutation/load enforce cross-version
@unique(typed literals for non-string columns); uniqueness coalesces by id (freed keys can be reused).open_dataset_at_stateis removed from the storage path. Tests/docs add regression coverage andmerge_costasserts validation opens only delta-touching tables.Reviewed by Cursor Bugbot for commit 268a5e9. Bugbot is set up for automated code reviews on this repo. Configure here.
Greptile Summary
This PR moves write constraint enforcement into one shared validator. The main changes are:
Confidence Score: 4/5
This is close, but the overwrite cardinality case should be fixed before merging.
crates/omnigraph/src/validate.rs
Important Files Changed
Comments Outside Diff (1)
crates/omnigraph/src/validate.rs, line 2108-2109 (link)When an overwrite replaces an edge table, this branch makes the committed cardinality view empty before deleted edge ids are resolved back to their old
srcvalues. The loader recordsdeleted_idsfor committed rows that are absent from the replacement image, butevaluate_cardinalitycallscommitted.committed_edges(edge_table, "id", &removed_ids)to find the sources those ids used to count against. For an overwritten table, that lookup returns no rows, so a replacement like committedAlice -> Acmebecoming onlyBob -> Acmechecks Bob but never checks Alice's final count of 0 against@card(1..). The overwrite can still commit a graph below the minimum cardinality bound.Reviews (6): Last reviewed commit: "docs(engine): refresh validate.rs module..." | Re-trigger Greptile
Context used: