From eef9f3e0765e089ad7a938be96d1285db128ab07 Mon Sep 17 00:00:00 2001 From: Gareth Price Date: Thu, 17 Sep 2026 22:44:18 +0100 Subject: [PATCH 01/22] docs(blog): rewrite build-system introduction from first principles --- docs/blog/01-why-sql-needs-a-build-system.md | 254 +++++++++++++++---- 1 file changed, 207 insertions(+), 47 deletions(-) diff --git a/docs/blog/01-why-sql-needs-a-build-system.md b/docs/blog/01-why-sql-needs-a-build-system.md index 5e9a853..f66a102 100644 --- a/docs/blog/01-why-sql-needs-a-build-system.md +++ b/docs/blog/01-why-sql-needs-a-build-system.md @@ -1,71 +1,231 @@ # 1. Why does SQL need a build system? -Start with one query. You have a table of raw events and you want a daily -count: +Start with the smallest possible data transformation: one SQL query. ```sql select occurred_at::date as day, kind, - count(*) as n + count(*) as events from landing.events group by 1, 2 ``` -Run it once in a console, paste the result into a dashboard, done. No build -system needed. So where does the complexity come from? +If you run that once in a console, look at the result, and throw it away, you do not need Phlo Transform. You do not need dbt either. You barely need a file. -## The moment it stops being one query +The interesting problems begin when the result stops being temporary. -The query becomes a system when three things happen: +## A query becomes a model when other things rely on it -1. **The result is stored.** Now it is a table, `marts.daily`, that other - queries read, and someone has to refresh it on a schedule. -2. **Other queries depend on it.** A weekly rollup reads `marts.daily`. A - churn model reads the rollup. You have a dependency graph whether you - wrote it down or not. -3. **The query changes.** Someone edits the definition. Which downstream - tables are stale now? Does a cosmetic edit justify a full rebuild? +Suppose we save the result as `marts.daily_events` because a dashboard needs it every morning. -None of these problems are about SQL. A compiler and a build system solve -the same problems for code: dependencies, incremental rebuilds, and knowing -what a change invalidates. A transformation layer is a build system whose -source language happens to be SQL. +Now we have two things: -## The accidental complexity tax +- a **definition**: the SQL above; +- a **materialisation**: the table or view produced from that definition. -Existing tools answer these questions, but with machinery on top: +That distinction is fundamental. The SQL is what we *want*. The relation in the warehouse is what currently *exists*. -- You declare dependencies with a special function, `ref('model')`, instead - of letting the tool read the `from` clause you already wrote. -- Models carry Jinja templates, so the file you edit is not the SQL that - runs. The file is a program that produces SQL, and you cannot know what it - does without executing it. -- Incremental behaviour lives in user-editable materialisation macros: - hundreds of lines of templated DDL per adapter. -- Correctness gates mostly do not exist. `dbt build` mutates production - directly, and you find out what it did by watching it. +A transformation tool has to keep those two things aligned. -Each item is a workaround for missing structure. A tool that understands -SQL can infer the dependency. A tool that owns materialisation does not -need a macro for incremental. A tool that records what it built can tell -you what a change invalidates before it touches anything. +Then another query reads `marts.daily_events`: -## What first principles look like +```sql +select + date_trunc('week', day) as week, + sum(events) as events +from marts.daily_events +group by 1 +``` + +We now have a dependency: + +```text +landing.events + │ + ▼ +marts.daily_events + │ + ▼ +marts.weekly_events +``` + +Nothing created that graph for us. It already existed the moment one query read another result. + +This is the first useful idea in Phlo Transform: + +> **The dependency graph is a property of the SQL, not of the tool.** + +The tool's job is to discover it accurately. + +## What is a relation? + +A SQL engine works with named things such as tables and views. In this series, we will use **relation** as the general term for one of those queryable objects. + +For example: + +```text +landing.events +marts.daily_events +analytics.customer_value +``` + +A relation may be an external input that Phlo does not build, or it may be the physical output of a model that Phlo does build. + +That gives us two more useful terms: + +- a **source** is an input relation that exists outside the transformation graph; +- a **model** is SQL owned by the workspace that produces a relation. + +If model B reads model A, A is upstream of B and B is downstream of A. + +## Why scheduling alone is not enough + +A simple scheduler could run model A and then model B every hour. That works until the system changes. + +Imagine someone reformats `daily_events.sql` without changing what it computes. Should the table rebuild? + +Now imagine they change: + +```sql +occurred_at::date +``` + +to: + +```sql +date_trunc('day', occurred_at at time zone 'UTC') +``` + +That may be a real semantic change. `daily_events` is now stale, and `weekly_events` may be stale too. + +Or perhaps the SQL is unchanged but the upstream source moved to a new Iceberg snapshot. The model definition did not change, but its input did. + +Or perhaps the table was modified outside the transformation tool after the last successful run. The tool's state says one thing while the warehouse holds another. + +A scheduler cannot answer these questions. It knows *when* to run commands, not *why* an output is valid. + +## This is a build-system problem + +Software build systems already solve the same class of problem. + +A compiler does not rebuild an entire program merely because a file's whitespace changed. It understands source files, dependencies, outputs and invalidation. + +A SQL transformation system needs the same concepts: + +1. **Source language** — the SQL files people write. +2. **Compiler** — parses those files and understands what they mean. +3. **Dependency graph** — which model reads which model or source. +4. **Build state** — what was previously materialised and from which inputs. +5. **Plan** — what needs to happen now, and why. +6. **Executor** — performs the required physical work. +7. **Evidence** — proves what was actually built and tested. + +The warehouse is not the build system. It is the execution target. + +## What does “materialise” mean? + +To **materialise** a model means to turn its SQL definition into a durable warehouse object. + +The simplest choices are: + +- **view** — store the query definition; compute rows when queried; +- **table** — run the query and store its result; +- **incremental table** — update only the part that needs changing. + +For a table model, a build might conceptually execute: + +```sql +create or replace table marts.daily_events as +select ... +``` + +For an incremental model it might perform an `INSERT`, `MERGE`, partition replacement or time-window update instead. + +The important point is that the model describes *what the data means*. The engine should own the mechanics of producing it. + +## Desired state and current state + +Phlo's planning model becomes much easier to understand if you separate two worlds. + +### Desired state + +The compiler calculates what each model should look like now: + +- its canonical SQL semantics; +- its configuration; +- its contract; +- the exact versions of upstream models; +- observed source states; +- the compiler semantics version. + +These inputs produce a content-addressed model version. + +### Current state + +The state store records what Phlo previously materialised: + +- model version; +- environment; +- physical target; +- adapter; +- producing run; +- materialisation timestamp; +- and, where the adapter can prove it, a strong physical output identity such as an Iceberg snapshot id. + +A plan is the comparison between desired and current state. + +That comparison produces a small vocabulary: + +```text +BUILD the required output is not currently proven to exist +SKIP this environment already has the required verified output +CACHED the required verified output already exists and can be adopted here +``` + +We will unpack all three later. For now, notice the important property: these are decisions backed by evidence, not merely commands. + +## The basic Phlo loop + +A local workspace can start with no external infrastructure: + +```bash +phlo-transform init +phlo-transform check +phlo-transform plan --adapter duckdb +phlo-transform run --adapter duckdb +``` + +`check` answers: + +> Can the workspace be compiled into a coherent model? + +`plan` answers: + +> Given the compiled workspace and the current state, what would happen? + +`run` performs the plan and then runs the relevant tests. + +The same compiler and engine can target Trino, Iceberg and Nessie for the lakehouse path. The execution substrate changes; the model of the workspace does not. + +## Why not “just run the SQL”? + +Because once transformations are shared, stored and changed over time, the hard questions are no longer SQL syntax questions: + +- What depends on this model? +- Which outputs became stale? +- Did the model change, or only its formatting? +- Did an input change? +- Does the physical table still match what we recorded? +- Can an existing output be reused safely? +- Which tests passed against the exact candidate we want to publish? +- Did production move after we audited it? +- What evidence authorised a promotion six months later? -Strip it back. A transformation workspace needs: +Those are build-system questions. -- **SQL files.** The source of truth, executable as written. -- **A compiler** that parses the files into a typed dependency graph. -- **State** that records what was built, so rebuilds stay minimal and - explainable. -- **A plan step before apply.** Every mutation follows an inspectable - decision. -- **Tests and audits inside the run**, not as a separate workflow. -- **Promotion gated on evidence**, not on hope. +Phlo Transform starts from that premise and tries to remove everything that does not follow from it. -That is Phlo Transform. The rest of this series walks through each piece -and shows the workflow running end to end, including a real dbt project -translated and executed on DuckDB. +The next step is the source language itself. If SQL already contains relation names and expressions, how much additional syntax does a compiler actually need? -*Next: [SQL should stay SQL](02-sql-should-stay-sql.md).* +*Next: [SQL should stay SQL](02-sql-should-stay-sql.md).* \ No newline at end of file From aac93817a9a451f3a6c2283a061a28b26606b7ae Mon Sep 17 00:00:00 2001 From: Gareth Price Date: Thu, 17 Sep 2026 22:44:47 +0100 Subject: [PATCH 02/22] docs(blog): explain the compiler from first principles --- docs/blog/02-sql-should-stay-sql.md | 314 +++++++++++++++++++++------- 1 file changed, 244 insertions(+), 70 deletions(-) diff --git a/docs/blog/02-sql-should-stay-sql.md b/docs/blog/02-sql-should-stay-sql.md index a3ba7f3..16bf17b 100644 --- a/docs/blog/02-sql-should-stay-sql.md +++ b/docs/blog/02-sql-should-stay-sql.md @@ -1,6 +1,8 @@ # 2. SQL should stay SQL -Here is a complete, working Phlo Transform model: +A build system needs a source language. For Phlo Transform, that language is SQL. + +Here is a complete model: ```sql select @@ -10,107 +12,279 @@ select from assay.raw_results ``` -No `{{ config() }}`. No `ref()`. No YAML file describing the model. No -schema declaration. This file is the model definition and the exact SQL -that executes. If `assay.raw_results` is another model in the workspace, -the compiler sees that and creates the dependency itself. +There is no `ref()` call declaring the dependency. There is no template layer producing another SQL string first. The file is SQL, and the SQL is the model definition. -## Why `ref()` exists at all +That sounds almost trivial, but it determines a large part of the architecture. -`ref('stg_orders')` does two jobs in dbt. It declares a dependency edge, -and it resolves the physical relation name. Both are workarounds. A SQL -parser already knows which relations a query reads, so -`from staging.stg_orders` *is* a dependency edge, written in a standard, -greppable way that every other tool understands. Physical naming is what a -compiler is for. +## What does a compiler actually do here? -The cost of `ref()` is more than aesthetics. Once dependencies go through a -template function, the file stops being SQL. No editor, linter, or query -engine can read it. Then every other feature piles onto the same templating -layer: macros, `var()`, `{{ this }}`, `is_incremental()`. The model becomes -a program you must execute to understand. +A compiler is often associated with turning a language such as Rust or C into machine code. More generally, a compiler takes one representation of a program, understands its structure, validates it, and produces a representation suitable for later stages. -## What inference looks like in practice +Phlo does the same thing with a transformation workspace. -Given this staging model: +A simplified pipeline is: -```sql --- transforms/staging/stg_orders.sql +```text +files + │ + ▼ +discovery + │ + ▼ +SQL parser + │ + ▼ +semantic models + │ + ├── relation resolution + ├── type/schema analysis + ├── dependency graph + ├── lineage + ├── contracts/tests + └── physical target resolution + │ + ▼ +compiled workspace +``` -select - id as order_id, - customer_id, - amount, - status, - ordered_at -from raw.orders +Execution comes later. The compiler should be able to answer as much as possible before a warehouse is mutated. + +## Step 1: discover models + +A workspace contains one or more transform roots. SQL files under those roots become candidate models. + +For example: + +```text +transforms/ +├── staging/ +│ ├── customers.sql +│ └── orders.sql +└── marts/ + └── customer_orders.sql +``` + +The path gives each file a stable logical identity: + +```text +staging.customers +staging.orders +marts.customer_orders ``` -Phlo reads `from raw.orders`, finds no model by that name in the workspace, -and records `raw.orders` as an external source. Downstream: +That logical identity is not necessarily the physical table name. It is the name used inside the workspace graph. + +This distinction matters because logical identity should remain meaningful even when an environment binds models to a different physical catalog. + +## Step 2: parse SQL, do not scan text + +Consider: ```sql --- transforms/marts/customer_orders.sql +with orders as ( + select * from raw.orders +) +select * from orders +``` + +A text search for `orders` cannot tell whether the final `from orders` means a workspace model, an external table or the CTE defined two lines earlier. +A SQL parser can. + +Phlo parses SQL into an abstract syntax tree. Relation extraction walks that tree with SQL scope rules, including CTE shadowing, aliases and nested queries. + +That is why dependency inference can be a compiler feature rather than a naming convention. + +## Step 3: resolve relation names + +Suppose we have: + +```sql +-- transforms/marts/customer_orders.sql select c.customer_id, - c.name, count(o.order_id) as order_count -from staging.stg_customers c -left join staging.stg_orders o +from staging.customers c +left join staging.orders o on o.customer_id = c.customer_id -group by c.customer_id, c.name +group by c.customer_id +``` + +The compiler sees two relation references: + +```text +staging.customers +staging.orders +``` + +It asks the workspace registry what each name means. + +If exactly one workspace model matches, the reference resolves to that model and creates a dependency edge. + +If no model matches, it is an external source. + +If more than one model could satisfy the name, compilation fails with an ambiguity diagnostic. + +The important rule is: + +> **Inference is only safe if ambiguity is an error.** + +Phlo does not “pick the most likely model” and hope. + +## Why no `ref()`? + +In dbt, this: + +```sql +from {{ ref('stg_orders') }} +``` + +both declares a dependency and asks the runtime to produce a physical relation name. + +Those are legitimate requirements, but they do not require a template function. + +The SQL already says which relation is read: + +```sql +from staging.orders +``` + +The compiler can infer the dependency from that syntax, and it can separately rewrite the logical name to the correct physical target after resolution. + +Keeping those responsibilities separate has a useful consequence: the model remains valid SQL throughout the process. + +## Logical names and physical targets + +A model might be written as: + +```sql +select * from staging.orders ``` -The two `staging.*` names resolve to models, and the edges appear. Output -of `phlo-transform list`: +while the physical relation used by a particular environment is: ```text -Models (5) - marts.customer_orders [table] marts.customer_orders <- staging.stg_customers, staging.stg_orders - marts.daily_revenue [incremental] marts.daily_revenue <- staging.stg_orders - marts.orders_incremental [incremental] marts.orders_incremental <- staging.stg_orders - staging.stg_customers [view] staging.stg_customers <- raw.customers - staging.stg_orders [view] staging.stg_orders <- raw.orders +phlo_ci_pr_42_ab12cd.analytics.staging__orders +``` + +The user should not have to write that physical name into the model. + +The compiler knows: + +1. `staging.orders` means a workspace model; +2. that model has a resolved physical target for this compilation; +3. the emitted SQL should reference that target. + +So model SQL is rewritten after semantic resolution. + +This is not string substitution. It is an AST-aware rewrite that follows the same scope rules as dependency extraction. + +## External sources use the same physical model + +Not every relation belongs to the workspace. -Sources (2) - raw.customers - raw.orders +```sql +select * from raw.instrument_results ``` -Each row shows the logical name, the materialisation, the physical target, -and what the model reads. +If `raw.instrument_results` is not a model, it becomes a source. + +When a default catalog is configured, Phlo qualifies unqualified or two-part external sources through that catalog. This matters under Nessie: a candidate environment uses a branch-bound catalog, and reads must resolve through the same environment as writes. -Names are logical. `staging.stg_orders` means "the model `stg_orders` in -the `staging` namespace", and the folder under `transforms/` provides the -namespace. Move the file and the identity moves with it. Physical naming is -the adapter's concern. +Without that rule, a candidate could write its own branch while accidentally reading a source through the session's base catalog. -## Inference has to be honest +A source already carrying an explicit catalog stays explicit. -Inference only works when the compiler refuses to guess. Two models that -could satisfy one relation name are an error: `check` fails and tells you. -Unparseable SQL is a diagnostic with a file and a position, not a fallback -to text matching. Ambiguity-as-error is what keeps plain SQL safe as the -dependency primitive. +The compiler and execution engine share the same source-to-relation mapping, so source probing, seed loading and emitted SQL agree on what physical relation a source means. -## What you give up, and what you get back +## Directives add intent without replacing SQL -No Jinja means no macro calls, no `env_var()` branching, no metaprogramming. -That constraint is real, and it is deliberate. A model that needs a -template engine to be read cannot be audited, diffed, or verified -statically. In exchange, every Phlo model parses, its graph is derivable, -and the compiler can answer questions like "what breaks if -`staging.stg_orders` changes?" without executing anything. +Some information is not present in a `SELECT` statement. -When a model does need directives, they live in comments. The file stays -valid to any SQL tool: +For example, SQL cannot tell us whether a model should be a table or view, or whether an incremental table should merge on `sample_id`. + +Phlo represents that information as configuration or lightweight comment directives: ```sql --- @incremental key=order_id --- @tags mart +-- @table +-- @key sample_id +-- @tags assay,validated select ... ``` -*Next: [One workspace, one graph](03-one-workspace-one-graph.md).* +or: + +```sql +-- @incremental key=sample_id +select ... +``` + +Comments preserve an important property: the file is still parseable SQL to editors and other tooling. + +The directives describe intent. They are not a second programming language. + +## Canonical semantics, not source bytes + +Once SQL is parsed, the compiler can reason about semantic structure rather than file text. + +That is the basis of content-addressed state later in the system. Reformatting this: + +```sql +select a,b from x +``` + +into this: + +```sql +select + a, + b +from x +``` + +should not invalidate data. + +A build system should care that the computation changed, not that the file's whitespace changed. + +## Compilation should fail early and specifically + +Because Phlo owns the compiler stage, invalid workspaces fail before execution. + +Typical errors include: + +- SQL parse failures; +- ambiguous model references; +- duplicate logical identities; +- physical target collisions; +- invalid directives; +- impossible contract declarations; +- dependency cycles. + +Diagnostics carry source locations where possible. A broken workspace does not become a half-executed run. + +This is one of the main reasons to keep SQL statically understandable: the tool can reject contradictions before they become warehouse mutations. + +## What is deliberately not supported? + +Plain SQL means giving up unrestricted compile-time metaprogramming inside models. + +Phlo does not try to execute arbitrary Jinja to discover what SQL might emerge. + +That is a constraint, but it buys strong properties: + +- dependencies are derivable without executing templates; +- models can be parsed and inspected directly; +- lineage can be calculated statically; +- plans are based on known semantics; +- migration tooling can clearly classify what is and is not translatable. + +The design principle is not “configuration is bad”. It is narrower: + +> **Do not add configuration for facts the compiler can safely infer from the program itself.** + +Once the compiler has resolved every model and source, the repository stops being a collection of SQL files. It becomes one semantic graph. + +That graph is the subject of the next post. + +*Next: [One workspace, one graph](03-one-workspace-one-graph.md).* \ No newline at end of file From 1ff3012d5b2eeff0d5c1b285792065798a17b200 Mon Sep 17 00:00:00 2001 From: Gareth Price Date: Thu, 17 Sep 2026 22:45:18 +0100 Subject: [PATCH 03/22] docs(blog): deepen workspace graph, typing, lineage and contracts --- docs/blog/03-one-workspace-one-graph.md | 339 ++++++++++++++++++++---- 1 file changed, 283 insertions(+), 56 deletions(-) diff --git a/docs/blog/03-one-workspace-one-graph.md b/docs/blog/03-one-workspace-one-graph.md index ef4064d..7c27eda 100644 --- a/docs/blog/03-one-workspace-one-graph.md +++ b/docs/blog/03-one-workspace-one-graph.md @@ -1,90 +1,317 @@ # 3. One workspace, one graph -dbt's unit of organisation is the project: one `dbt_project.yml`, one -namespace, with other projects bolted on through a package mechanism. -Phlo's unit is the workspace, the whole repository, and it dissolves the -project boundary on purpose. +Once every SQL file has been parsed and every relation reference resolved, the repository stops being a pile of queries. -## Discovery instead of declaration +It becomes a graph. -A Phlo workspace finds models wherever they live: +That graph is the central data structure behind most of Phlo Transform: build order, selection, lineage, impact analysis, state propagation, tests, contracts and promotion all depend on it. + +## What is a graph? + +A graph is just a set of things connected by relationships. + +For transformations, the things are models and sources. The relationships are dependencies. + +If `assay.results` reads `assay.raw`, we draw: ```text -transforms/ # shared, repo-level transforms -workflows/assay/transforms/ # transforms owned by a workflow -workflows/reporting/transforms/ +assay.raw ─────► assay.results ``` -Drop a `.sql` file into any of those roots and the file joins the graph. -Same compiler, same rules, same namespaces. A model under -`workflows/assay/transforms/` belongs to the `assay` workflow and to the -one workspace graph, so `reporting.monthly` can depend on `assay.results` -with an ordinary `from` clause: +If a report then reads `assay.results`: ```text -$ phlo-transform list - -Models (3) - assay.results [table] <- assay.raw - assay.raw [table] - reporting.monthly [view] <- assay.results +assay.raw ─────► assay.results ─────► reporting.monthly ``` -No package installation, no cross-project `ref`, no import step. The graph -is derived from the files that exist. +The direction means “this output depends on that input”. -## Why one graph matters +A transformation graph is normally a **directed acyclic graph**, or DAG: -Split the graph and every downstream feature gets worse: +- **directed** because dependency has a direction; +- **acyclic** because a model cannot ultimately depend on itself. -- **Lineage stops at the boundary.** "What consumes `assay.results`?" turns - into a question for three tools and a wiki page. -- **Rebuild planning cannot see upstream.** An upstream project rebuilt. - Downstream staleness gets handled by convention and cron ordering. -- **Tests cannot gate across the boundary** without orchestration glue. +A cycle such as A → B → A is not an execution-order problem to solve later. It is a contradiction in the workspace, so compilation fails. -One graph means `lineage`, `impact`, `plan` and `test` all see the same -truth: +## Why the whole workspace should be one graph + +Repositories are often organised for people rather than compilers: ```text -$ phlo-transform lineage reporting.monthly -Model: reporting.monthly -Upstream: assay.results, assay.raw -Downstream: (none) +transforms/ +workflows/assay/transforms/ +workflows/reporting/transforms/ ``` +Those folders can represent ownership, domains or application boundaries. They do not need to become separate transformation universes. + +Phlo discovers multiple transform roots and compiles them into one logical workspace. + +For example: + +```text +workflows/assay/transforms/raw.sql +workflows/assay/transforms/results.sql +workflows/reporting/transforms/monthly.sql +``` + +can become: + ```text -$ phlo-transform impact assay.raw +assay.raw + │ + ▼ assay.results + │ + ▼ reporting.monthly ``` -## The graph is typed, not textual +`reporting.monthly` can simply contain: -The compiler resolves columns and types, so `explain` reports more than -edges. `marts.daily` reads `staging.stg_events`, and its `day` column -derives from `staging.stg_events.occurred_at` as a `DATE`: +```sql +select * from assay.results +``` + +No package mechanism is required to make the dependency visible because the compiler already sees both sides. + +Ownership remains useful metadata. Fragmenting the graph is not required to express it. + +## Why graph boundaries are expensive + +Suppose `assay.results` changes. + +A useful system should be able to answer: + +- Which models are downstream? +- Which columns are affected? +- Which tests may need to run again? +- Which planned outputs are stale? +- Which candidate data should be compared with production? + +If the workspace is split into disconnected project graphs, each question has to cross those boundaries using extra manifests, package metadata or orchestration glue. + +One graph means the same compiler truth feeds every feature. + +## Topological order: how the graph becomes an execution order + +A graph says what depends on what. The runner still needs an order that respects those dependencies. + +For: ```text -Columns (3): - day DATE unknown <- staging.stg_events.occurred_at - kind VARCHAR unknown <- staging.stg_events.kind - n BIGINT not null +A ──► B ──► D +└──► C ──► D ``` -Column-level truth is what powers the generated tests, the schema -contracts, and column-level impact analysis. `@key` on a model implies -unique and not-null assertions compiled to real SQL tests. +A valid order might be: + +```text +A, B, C, D +``` + +or: + +```text +A, C, B, D +``` + +B and C can run concurrently after A finishes. D must wait for both. + +That is a **topological order**: an ordering where every node appears after its dependencies. + +Phlo's runner uses the graph directly rather than turning the whole workspace into one long serial list. Independent branches can execute concurrently as soon as their upstream work is satisfied. + +We will return to the scheduler later. + +## A useful graph contains more than model names + +A dependency graph that only knows “A depends on B” is valuable, but limited. + +Phlo's compiler builds a semantic model of the query too. + +Consider: + +```sql +select + experiment_id, + result * dilution_factor as corrected_result +from assay.raw_results +``` + +At model level we know: + +```text +assay.raw_results ──► assay.corrected_results +``` + +At column level we can know that: + +```text +assay.corrected_results.experiment_id + ← assay.raw_results.experiment_id + +assay.corrected_results.corrected_result + ← assay.raw_results.result + ← assay.raw_results.dilution_factor +``` + +That is **column lineage**. + +## Lineage and impact are inverse questions + +Lineage asks: + +> Where did this thing come from? + +Impact asks: + +> What depends on this thing? -The compiler also needs the catalogue for full fidelity. Offline, an -external source's column types come back `unknown`. Pass `--adapter` and -Phlo enriches the types from the real schema. +For a model: + +```bash +phlo-transform lineage reporting.monthly +``` + +can walk upstream. + +For a model or column: + +```bash +phlo-transform impact assay.results +``` + +can walk downstream. + +Because both are derived from the compiler graph, they are not separate metadata that can drift from execution. + +Phlo can also export the canonical graph as OpenLineage-compatible events for systems that consume that standard. + +## Where types come from + +SQL expressions have types: integer, decimal, varchar, timestamp, boolean and so on. + +Some types are inferable entirely from SQL syntax. Others depend on the schema of an external relation. + +For example: + +```sql +select + sample_id, + concentration * dilution as corrected +from raw.assay_results +``` + +The compiler needs to know what `concentration` and `dilution` are to infer the output precisely. + +Offline, external source columns may be unknown. + +With an adapter configured, Phlo can ask the real catalogue for relation columns and enrich the compilation. + +This is an important boundary: + +```text +compiler truth available from files + + +optional observed warehouse schema + = +richer semantic model +``` + +The compiler remains separate from the warehouse adapter; enrichment is an input, not a hidden execution side effect. + +## What is a contract? + +A contract is an explicit promise about a model's output schema. + +For example, a downstream application may rely on: + +```text +sample_id VARCHAR NOT NULL +result DOUBLE +run_date DATE +``` + +If a model silently removes `sample_id` or changes it to an incompatible type, the SQL might still execute successfully while consumers break. + +A schema contract turns that implicit dependency into something the system can check. + +Contracts become part of the model version and part of promotion auditing. A breaking contract change can block promotion unless it is explicitly allowed. + +Declared renames can distinguish: + +> this column disappeared + +from: + +> this column intentionally moved from `old_name` to `new_name`. + +That is more useful than comparing opaque schema hashes. + +## Keys are semantic information too + +A key says which columns identify a row. + +```sql +-- @key experiment_id,sample_id +select ... +``` + +For a composite key, the pair is unique; the individual columns do not need to be unique by themselves. + +Phlo turns the declaration into several consistent behaviours: + +- not-null assertions for the key columns; +- one composite uniqueness assertion; +- row identity for keyed diffs; +- merge identity for keyed incremental models where applicable. + +One declaration drives several features because they are all asking the same semantic question: *what identifies a row?* + +## Tests are nodes around the graph, not comments in documentation + +A generated or custom test is executable SQL with declared targets and sources. + +Examples include: + +- `not_null`; +- `unique`; +- accepted values; +- custom SQL assertions. + +Tests run against actual materialised outputs. A model can execute successfully and still fail its tests, which causes the overall run to fail without pretending the SQL execution itself failed. + +That distinction matters later for retries and promotion evidence. + +## Multi-root does not mean “anything can depend on anything” + +A single graph is not the same as having no boundaries. + +Ownership and policy can still restrict which namespaces or workflows may consume others. The important architectural choice is that those rules operate *on one known graph* rather than hiding edges behind separate build systems. + +This gives us a clean mental model: + +```text +repository + │ + ▼ +compiler + │ + ▼ +one typed graph + │ + ├── build order + ├── lineage + ├── impact + ├── tests + ├── contracts + ├── version propagation + └── promotion analysis +``` -## Ownership without fragmentation +The next question is what happens over time. Once yesterday's graph has produced physical tables and today's graph has changed, how does Phlo decide what to build, skip or safely reuse? -Workflows still own their folders. `workflows/assay/` is the assay team's -directory, and workspace policy can gate which workflows may consume which -namespaces. Ownership is metadata on one graph, not a wall between graphs. -That is the balance: monorepo coherence with per-team boundaries. +That is a state and planning problem. -*Next: [Plan before you apply](04-plan-before-apply.md).* +*Next: [Plan before you apply](04-plan-before-apply.md).* \ No newline at end of file From 3397124d46ec207b20aca5f4f76ebbb85ce175b5 Mon Sep 17 00:00:00 2001 From: Gareth Price Date: Thu, 17 Sep 2026 22:45:58 +0100 Subject: [PATCH 04/22] docs(blog): explain state, planning and verified cache reuse --- docs/blog/04-plan-before-apply.md | 454 +++++++++++++++++++++++++----- 1 file changed, 384 insertions(+), 70 deletions(-) diff --git a/docs/blog/04-plan-before-apply.md b/docs/blog/04-plan-before-apply.md index 2b6e63e..193be34 100644 --- a/docs/blog/04-plan-before-apply.md +++ b/docs/blog/04-plan-before-apply.md @@ -1,100 +1,414 @@ # 4. Plan before you apply -The most expensive habit in data tooling is "run first, ask questions -later". `dbt build` against production tells you what it is doing only by -doing it. Phlo splits every mutation into a plan that explains itself and -an apply that executes what the plan said. +A compiler tells us what the workspace means now. A warehouse contains whatever was built in the past. -## What a plan looks like +A transformation engine needs to reconcile those two worlds without guessing. + +That is the purpose of **state** and **planning**. + +## Start with the simplest question + +Suppose this model exists: + +```sql +select + sample_id, + result +from assay.raw_results +``` + +Yesterday, Phlo built it successfully. + +Today, should it run again? + +A timestamp is not enough to answer that. Neither is “the file changed” or “the table exists”. + +The real question is: + +> **Does the physical output we have now still prove that it represents the model we want now?** + +Phlo answers that question before execution. + +## What is state? + +State is a durable record of what Phlo has observed and produced. + +The default local store is SQLite: + +```text +.phlo/transform/state.db +``` + +A shared Postgres backend is also available for workflows where different CI stages or machines need to see the same run and audit history. + +State contains things such as: + +- runs and their status; +- per-model execution results; +- stored plans; +- model versions that were materialised; +- the physical target used; +- adapter identity; +- time-window watermarks; +- seed versions; +- promotions; +- portable audit evidence. + +For adapters that can prove physical identity, a materialised record also carries an **output identity**. + +For Iceberg, that is the snapshot id. + +## A model version is a statement about meaning + +Phlo computes a content-addressed version for every compiled model. + +The version incorporates the pieces that can change what the model computes: + +```text +canonical SQL semantics +configuration +contract +dependency versions +source states +content slot/materialisation +compiler semantics version +``` + +A few details are worth unpacking. + +### Canonical SQL semantics + +The version is derived from parsed/canonical semantics rather than source bytes, so harmless formatting does not create a new data version. + +### Dependency versions + +If an upstream model changes, a downstream model's desired version changes even if the downstream SQL file itself is untouched. + +That is how invalidation propagates through the graph. + +### Source states + +External inputs can move without SQL changing. + +An Iceberg source can be represented by a snapshot identity. DuckDB uses a more limited schema/row-count fingerprint. If an observed source state moves, models that consume it become stale. + +### Compiler semantics version + +Sometimes the compiler itself changes how the same source program is interpreted. Phlo carries an explicit compiler-semantics version so an upgrade can trigger a one-time honest rebuild rather than silently treating old and new semantics as equivalent. + +### Why the catalogue is not in the model version + +This became important for v0.1. + +Consider the same logical model materialised through two Nessie-bound catalogs: + +```text +phlo_main.analytics.assay__results +phlo_ci_42.analytics.assay__results +``` + +The catalog says **where an environment sees the model**. It does not change what the model computes. + +So the version hashes the content slot — effectively `schema.table|materialisation` — rather than the environment's catalog binding. + +Physical target movement is still detected separately. Removing the catalog from semantic identity does not mean Phlo ignores where data lives. + +## Desired state versus recorded state + +For each model, the planner has two sides. + +### Desired + +From the current compilation: + +```text +version = abc123 +physical target = phlo_ci_42.analytics.assay__results +``` + +### Recorded + +From state: + +```text +version = abc123 +physical target = phlo_main.analytics.assay__results +adapter = trino +output identity = snapshot:918273645 +``` + +It also asks the adapter what exists *right now*. + +That last step is critical. State is evidence from the past. The warehouse is allowed to have changed since then. + +## The three normal actions + +Phlo plans every model as one of three meaningful actions. + +### `BUILD` + +Phlo cannot prove the required output already exists safely. + +Reasons include: + +- target relation missing; +- SQL semantics changed; +- config changed; +- contract changed; +- upstream dependency changed; +- source state changed; +- adapter changed; +- output identity drifted; +- physical target moved and the old output is not verifiably present there; +- `--force` was requested. + +### `SKIP` + +This environment's own record already vouches for the desired version at this target, and any strong physical identity still matches live. + +Nothing needs to execute. + +### `CACHED` + +The desired output was recorded elsewhere, and this environment can prove it already sees the same physical output. + +This is not “we found the same hash in a database, so let's trust it”. It is **verified adoption**. + +That distinction deserves its own section. + +## What does cache reuse mean for data? + +In a traditional build cache, a compiler may copy a previously built artifact into the current build. + +A lakehouse can sometimes do something better: the target environment already sees the same immutable physical object. + +Nessie makes this common. + +Imagine `main` contains an Iceberg table at snapshot `S1`: ```text -$ phlo-transform plan --adapter duckdb --duckdb-path shop.duckdb +main +└── assay.results → snapshot S1 +``` + +Create a candidate branch from `main`: + +```text +main candidate +└── results S1 └── results S1 +``` + +Before the candidate changes anything, both references can see the inherited snapshot. + +If the compiled candidate wants the same model version, Phlo can ask: + +1. Is there a materialised record for this exact model version elsewhere? +2. Was it produced by the same adapter semantics? +3. Does it describe the same content slot? +4. Does it carry a strong output identity? +5. Does the candidate's own target report that same identity live? + +If all of those are true, the candidate does not need to run model SQL. + +It can adopt the output. + +## Adoption is an execution action + +`CACHED` is not just a planner label. + +The plan carries provenance for the reusable output: + +```text +source environment +source physical target +output identity +producing run id +materialised-at timestamp +``` + +Immediately before adoption, the runner asks for the target's live output identity **again**. + +Why twice? + +Because time exists between planning and execution. + +A table could change after the plan was created. + +If the identity no longer matches, the runner changes course and performs a full build with a `cache_miss` reason. It does not adopt stale evidence. + +If the identity still matches, Phlo writes an environment-local materialisation record pointing at the verified output. + +It preserves the original producing run id and materialisation timestamp. Reuse must not pretend that this run physically created data it did not create. + +For time-window incrementals, the matching watermark is adopted too because identical content has the same processing frontier. + +After adoption, the candidate has its own state record, so the next no-op plan is a normal `SKIP` rather than another cross-environment cache lookup. + +## Physical identity is stronger than “the table exists” + +Consider this failure mode: + +```text +state says assay.results = version abc123 +warehouse table exists +someone rewrites the table outside Phlo +``` + +If the planner checked only the relation name, it could incorrectly skip. + +For Iceberg, Phlo records the snapshot id after a successful build and checks it again on later plans. + +If state says: + +```text +snapshot:100 +``` + +and live storage says: + +```text +snapshot:104 +``` -Models (5) - BUILD staging.stg_customers [view] staging.stg_customers - reason: target relation does not exist - BUILD staging.stg_orders [view] staging.stg_orders - reason: target relation does not exist - BUILD marts.customer_orders [table] marts.customer_orders - reason: target relation does not exist - BUILD marts.daily_revenue [incremental] marts.daily_revenue - reason: target relation does not exist - strategy: time-window - full rebuild required - BUILD marts.orders_incremental [incremental] marts.orders_incremental - reason: target relation does not exist - strategy: key - full rebuild required +then the record no longer proves the table contains Phlo's recorded output. + +The model rebuilds. + +This is why warm planning is not free. At large scale, strong drift checks require real metadata reads. In v0.1, a 5,000-model warm Trino/Iceberg plan is dominated by per-table snapshot lookups. Phlo deliberately pays that cost rather than turning “probably unchanged” into `SKIP`. + +## What does a plan look like? + +A first run might produce: + +```text +BUILD staging.customers + target relation does not exist + +BUILD staging.orders + target relation does not exist + +BUILD marts.customer_orders + target relation does not exist +``` + +After a successful run: + +```text +SKIP staging.customers + SQL, config, contract and inputs unchanged + +SKIP staging.orders + SQL, config, contract and inputs unchanged + +SKIP marts.customer_orders + SQL, config, contract and inputs unchanged +``` + +On a fresh Nessie candidate inheriting verified base outputs: + +```text +CACHED staging.customers + candidate target holds the desired snapshot recorded in main; adopting it ``` -Every model carries an action, `build`, `skip`, or `cached`, and the -reason for the action. The plan is written to `.phlo/` as a versioned -artifact, so the decision is an auditable record and not console output. -`plan --json` gives the same record to machines. +The JSON plan carries the same structured reasons for automation. + +## Plan first means apply does not re-decide intent + +This separation is subtle but important. -A workspace that does not compile blocks the plan and exits non-zero. -Nothing executes against a broken graph: +The planner decides *what should happen*. + +The runner decides *how to execute that plan safely*. + +The runner can reject a stale plan or downgrade an unsafe cache adoption to a build, but it does not silently invent a different desired model graph. + +A plan can therefore be reviewed by a human or machine before mutation. + +`run` is the convenient combined workflow: ```text -plan blocked by compilation errors - UNKNOWN marts.labelled [table] marts.labelled - ... -error[PARSE001]: could not parse model SQL ... --> transforms/marts/labelled.sql +compile → plan → apply → tests ``` -## How "what changed" gets computed +but the boundaries still exist internally. -Each compiled model gets a content-addressed version: a hash over the -canonical SQL semantics, the config, the contract, the dependency versions, -the source states, and the compiler semantics version. After a run, the -materialised version lands in local state at `.phlo/transform/state.db`. +## Stale plans are rejected -The next `plan` is a comparison, not a guess: +Suppose a plan says: ```text -$ phlo-transform run # second time, nothing changed +model: assay.results +target: catalog_a.analytics.assay__results +version: abc123 +``` + +Then the workspace is recompiled against `catalog_b` before apply. + +Because catalog identity is intentionally outside the semantic version hash, `abc123` may still match. + +That does **not** make the old plan valid. + +Plan staleness checks the full physical target too. A plan inspected against one target cannot silently execute against another. + +The same principle applies when the model's desired version changes after planning. - SKIP staging.stg_customers [view] staging.stg_customers - SKIP staging.stg_orders [view] staging.stg_orders - SKIP marts.customer_orders [table] marts.customer_orders - ... +## `--since` reduces the problem before planning it + +Sometimes we already know which source files changed relative to Git. + +```bash +phlo-transform plan --since main ``` -Each change kind has its own reason. Edit the SQL and the reason is `SQL -semantics changed`. Edit config and the reason is `config changed`. A -source moves and the reason is `a source state changed`: on Iceberg the -state is a snapshot id, on DuckDB a schema and row-count fingerprint. A -dependency's version moves and the reason is `an upstream model version -changed`, propagated transitively. +Phlo maps the Git change set back into model identity, expands the necessary dependency closure, and plans only the affected subset. + +This matters at scale because expensive physical verification is only needed for the selected work. -Two properties matter here: +The graph and state system cooperate: + +```text +Git change + │ + ▼ +changed model set + │ + ▼ +dependency/impact closure + │ + ▼ +state-aware plan +``` -- **The hash is semantic, not textual.** Reformatting a file does not - rebuild data, because the version covers canonicalised SQL rather than - bytes. -- **Stale plans are rejected.** A plan binds to the state it was computed - against. If the world moved on, you plan again. Apply never executes a - decision made against a different reality. +## `inspect` is the same accounting at one-model scale -## `inspect` shows the same accounting per model +For one model, the useful view is: ```text -State: - desired: 8974fcb - current: 8974fcb - status: unchanged +Desired version: abc123 +Current version: abc123 +Status: unchanged +Target: ... +Adapter: ... +Output identity: ... ``` -Desired is what the compiler wants now. Current is what is materialised in -this environment. That one comparison, computed instead of assumed, is the -core of state-aware execution. +The point is not the formatting. It is that `inspect`, `explain`, `plan` and `run` all derive from the same compiler and state model rather than each inventing their own definition of “changed”. + +## Why this matters beyond performance + +Minimal rebuilds are useful, but the deeper benefit is accountability. + +For every model, Phlo can explain: + +- what it wants; +- what it previously built; +- what it can currently verify; +- which input moved; +- why the chosen action follows. -## Why the split pays for itself +That foundation is what makes incrementals, branch isolation and promotion evidence tractable. -`run` is a convenience wrapper for plan, apply, and tests. The split is -what the rest of this series builds on: promotion gates that require a -green run on a candidate branch, data diffs computed before merge, and CI -that fails on a plan which would rebuild production. All before a single -byte is written. +The next post looks at incremental materialisation specifically: how to update only part of a table without putting control-flow programs inside model SQL. -*Next: [Incremental models without macros](05-incremental-without-macros.md).* +*Next: [Incremental models without macros](05-incremental-without-macros.md).* \ No newline at end of file From b37bf7aabc04598419407c3622bf1545279d7b1a Mon Sep 17 00:00:00 2001 From: Gareth Price Date: Thu, 17 Sep 2026 22:46:29 +0100 Subject: [PATCH 05/22] docs(blog): deepen incremental model explanation --- docs/blog/05-incremental-without-macros.md | 391 +++++++++++++++++---- 1 file changed, 329 insertions(+), 62 deletions(-) diff --git a/docs/blog/05-incremental-without-macros.md b/docs/blog/05-incremental-without-macros.md index ee44090..c677dad 100644 --- a/docs/blog/05-incremental-without-macros.md +++ b/docs/blog/05-incremental-without-macros.md @@ -1,30 +1,95 @@ # 5. Incremental models without macros -Incremental models are where transformation tools earn or lose their keep. -A full rebuild is always correct and always expensive. An incremental run -is cheap and correct only when the machinery underneath knows exactly what -"new data" means for that model. +A table model is easy to reason about if you rebuild the whole table every time: -In dbt, the machinery is yours. You write `{{ config(materialized = -'incremental') }}`, then guard the expensive part by hand with -`{% if is_incremental() %} ... where ts > (select max(ts) from {{ this }}) -{% endif %}`, and the adapter's materialisation macro decides what DDL to -emit. The intent, "merge by key" or "append rows past a watermark", is -buried in templated control flow. +```text +old table + │ + └── replace with result of current query +``` + +That is simple and usually correct. It can also be wasteful. + +If a ten-billion-row event table receives one million new rows, rebuilding all ten billion rows to incorporate that million may be unnecessary. + +An **incremental model** updates only the part that needs changing. + +The hard part is not performance. The hard part is preserving correctness while doing less work. + +## Incremental is not one algorithm + +“Incremental” describes a goal, not an implementation. + +Different data has different update semantics. + +### Append + +New rows only arrive at the end. + +```text +existing rows + new rows +``` + +### Keyed update + +A row has an identity such as `sample_id`, and newer input may replace the previous value for that identity. + +```text +sample_id=42, result=1.2 + ↓ +new input: sample_id=42, result=1.4 + ↓ +replace/update row 42 +``` + +### Partition replacement + +A bounded partition such as `run_date=2026-09-17` can be recomputed as a unit without touching other partitions. -## Declaring intent instead +### Time window -Phlo makes incremental a declaration on the model. Four strategies cover -the common shapes: +Rows can be selected after a known processing frontier, usually with some overlap to catch late-arriving data. + +These are different operations. Treating all of them as one “incremental mode” pushes the complexity somewhere else. + +## Why Phlo declares the strategy + +In a templated transformation system, incremental behaviour often lives inside the model itself: + +```text +if incremental: + add this filter +else: + run something else +``` + +Now the SQL has two possible meanings depending on runtime state. + +Phlo instead makes the incremental **strategy** explicit configuration. + +Examples: ```sql -- @incremental append +select ... +``` + +```sql -- @incremental key=experiment_id +select ... +``` + +```sql -- @incremental partition=run_date +select ... +``` + +```sql -- @incremental window=updated_at +select ... ``` -Or the same in `phlo.toml`: +A time-window model can also be configured with an overlap: ```toml [model.events.incremental] @@ -33,53 +98,255 @@ column = "updated_at" overlap = "2h" ``` -The strategy becomes part of the model's config, which makes it part of the -model's version. Change the key and the plan notices: +The SQL still defines the desired rows. The strategy tells the engine how an existing materialisation may be updated safely. + +## Strategy is part of model identity + +Changing: + +```text +append +``` + +to: ```text - BUILD marts.orders_incremental [incremental] marts.orders_incremental - reason: config changed - strategy: key - full rebuild required -``` - -## What the engine does with the declaration - -- **Bootstrap or full rebuild:** `create table ... as`, or the equivalent - replace. -- **append:** `insert into