Skip to content

feat: semantic lineage diff across Git refs (Bundle 8) - #12

Merged
iamgp merged 1 commit into
feat/contracts-breakingfrom
feat/lineage-diff
Sep 15, 2026
Merged

iamgp merged 1 commit into
feat/contracts-breakingfrom
feat/lineage-diff

Conversation

@iamgp

@iamgp iamgp commented Sep 13, 2026

Copy link
Copy Markdown
Contributor

Summary

Bundle 8 of the lakehouse-native stack, based on Bundle 7 (#11). lineage --diff <git-ref> is the code-side complement to the data-side branch_diff: it answers "what does this branch change about the graph itself" before promote merges the data branch.

  • core/lineage_diff.rs: lineage_diff(base, candidate) compares two LineageGraphs node-for-node and edge-for-edge — nodes_added/removed/changed (version, materialization, target, type, path), edges_added/removed (dependency structure — a model that started reading a dataset is a changed input edge, not just a changed file), edges_changed (column derives edges whose transformation/directness/confidence/expression moved), and impacts naming the base-side consumers each removed node orphans.
  • core/git.rs::checkout_tree: materialises the workspace subtree at a resolved commit into a tempdir via ls-tree + cat-file --batch — read-only, no worktree, safe beside a dirty checkout or in-progress rebase. Handles nested workspace prefixes and symlinked files.
  • lineage --diff <ref>: loads + compiles the base tree, applies the same --catalog override and catalogue enrichment as the candidate compile (so enriched source columns do not diff as "added"), prints the delta, writes .phlo/transform/lineage_diff.json with the resolved base_commit, and fails cleanly when the base ref cannot load or compile rather than diffing a partial graph.
  • NodeMeta gains PartialEq for field-level comparison.

In the WAP flow the two reviews now sit side by side: lineage --diff proves the definitions, branch_diff proves the data.

Test plan

  • cargo test --workspace — all suites green
  • cargo clippy --workspace --all-targets -- -D warnings — clean
  • cargo fmt --all -- --check — clean
  • 6 unit tests in lineage_diff.rs (empty diff, added/removed, version change, new dependency edge, orphan impact, column-lineage change)
  • 4 CLI tests against a real git repo (clean diff, removed model + orphans, changed model JSON, artifact written)

Stacked on #11 — do not merge independently.

@iamgp
iamgp force-pushed the feat/lineage-diff branch 2 times, most recently from c6e5465 to dca0215 Compare September 14, 2026 09:53
@iamgp
iamgp force-pushed the feat/lineage-diff branch 2 times, most recently from 8b4bde4 to dc9438b Compare September 14, 2026 17:57
…ee compilation, orphaned-consumer impact

Bundle 8 on top of contracts/breaking. `lineage --diff <git-ref>` is the
code-side complement to the data-side `branch_diff`: it answers "what
does this branch change about the graph itself" before promote merges
the data branch.

- core/lineage_diff.rs: lineage_diff(base, candidate) compares two
  LineageGraphs node-for-node and edge-for-edge — nodes added/removed/
  field-changed (version, materialization, target, type, path), input
  edges gained or lost (dependency structure, not just files), column
  derives edges whose transformation/directness/confidence/expression
  moved, and per removed node the base-side consumers it orphans.
- core/git.rs checkout_tree: materialises the workspace subtree at a
  resolved commit into a tempdir via ls-tree + cat-file --batch —
  read-only, no worktree, safe beside a dirty checkout. Symlinks and
  nested workspace prefixes handled.
- lineage --diff <ref>: loads + compiles the base tree, applies the same
  catalog override and catalogue enrichment as the candidate compile so
  both sides compare like for like, prints the delta, writes
  lineage_diff.json with the resolved base_commit, and fails cleanly
  when the base ref cannot load or compile.
- NodeMeta gains PartialEq for field-level comparison.
@iamgp
iamgp added this pull request to stack #15 September 15, 2026 06:48
@iamgp
iamgp merged commit 26ac6c5 into main Sep 15, 2026
1 check passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant