Status: Implemented
Language: Rust
Primary execution target: Trino + Iceberg + Nessie
Secondary targets: DuckDB, PostgreSQL
Primary host: Phlo
CLI namespace: phlo-transform
Phlo Transform is a workspace-native SQL transformation compiler and execution engine.
It provides the useful core capabilities associated with tools such as dbt while deliberately avoiding much of their configuration surface, runtime templating complexity, project isolation, adapter abstraction complexity, and legacy compatibility requirements.
Phlo Transform is designed around five ideas:
- SQL should remain SQL.
- The compiler should infer what it can.
- The whole repository is one workspace.
- Transforms are compiled into a typed dependency graph before execution.
- Changes are planned, audited and promoted rather than blindly executed.
A minimal model should therefore be valid as:
select
experiment_id,
sample_id,
result
from assay.raw_resultsIf assay.raw_results is another model in the workspace, Phlo Transform automatically creates the dependency.
No ref() is required. No model YAML is required. No source declaration is required. No explicit DAG is required. No schema declaration is required unless the user wants a contract stricter than the inferred schema.
The same compiler representation powers:
- model execution;
- dependency resolution;
- model selection;
- type checking;
- schema validation;
- column lineage;
- impact analysis;
- incremental execution;
- state comparison;
- data diffing;
- workflow integration;
- AI/agent interfaces;
- documentation;
- auditability.
Phlo Transform is therefore not intended to be dbt rewritten in Rust. It is intended to be a Rust-native SQL build system for modern data workspaces.
Phlo Transform must:
- execute SQL transformation DAGs;
- discover transforms across multiple folders;
- support transforms local to workflows;
- construct one workspace-wide dependency graph;
- understand SQL structurally rather than primarily as text;
- infer dependencies directly from SQL;
- infer external sources where possible;
- minimise configuration;
- support table, view and incremental materialisations;
- support static and runtime data-quality rules;
- support schema contracts;
- support state-aware execution;
- detect changes before execution;
- support plan/apply semantics;
- integrate natively with Nessie branches;
- support Write-Audit-Publish workflows;
- provide data and schema diffs;
- expose model and column lineage;
- expose machine-readable APIs for agents and UIs;
- integrate transformation graphs into wider Phlo workflow graphs;
- be embeddable as a Rust library;
- have predictable, deterministic behaviour;
- produce reproducible execution artifacts.
Initial versions will not attempt to provide:
- complete dbt compatibility;
- arbitrary dbt macros;
- arbitrary Jinja execution;
- arbitrary Python execution inside model definitions;
- dbt package compatibility;
- every database adapter;
- semantic-layer functionality;
- BI metrics definitions;
- dashboard authoring;
- notebook functionality;
- orchestration of arbitrary non-transform workloads;
- hundreds of configuration precedence behaviours;
- adapter macro dispatch;
- user-defined materialisation code;
- Python models;
- dbt exposures;
- dbt snapshots;
- dbt seed compatibility;
- a complete dbt-style documentation website.
These should be introduced later only where there is a clear need.
A valid transformation should usually require only a .sql file. Configuration exists to express exceptions, not basic operation.
Avoid situations where the same setting can be specified through several unrelated configuration paths. Configuration precedence must remain intentionally small.
The majority of transformation files should contain valid SQL.
Preferred:
select *
from assay.resultsrather than:
select *
from {{ ref("results") }}The compiler should understand the project before running it. Where feasible, errors should be caught at compile or plan time rather than at warehouse execution time.
A repository is one transformation workspace. Transforms may live anywhere within defined transform roots. Workflows can own local transform trees without becoming isolated projects.
Phlo Transform must never resolve ambiguous references silently. Ambiguous names are compilation errors.
Previous execution state is part of the execution model rather than an optional optimisation layer.
Any potentially meaningful change should be inspectable before execution or promotion.
Logical model identity must not depend entirely on filesystem location.
Anything the CLI can explain should also be available in structured form. Agents and UIs should never have to scrape human-readable terminal output.
Every feature should first ask whether the compiler can infer the required information safely. Only require configuration when inference would be unreliable or ambiguous.
A repository or logical project containing transform roots and optionally workflows.
A directory containing transformation models, for example:
transforms/shared/
workflows/assay_ingest/transforms/
workflows/manufacturing/transforms/
A logical grouping used for model identity and reference resolution, for example assay_ingest.
A SQL transformation producing a logical dataset.
A relation referenced by models but not produced within the current workspace.
An immutable representation of a model at a particular code, configuration and dependency state.
How the logical model is physically represented. Initially: view, table, and incremental.
Assertions about model schema and structural behaviour.
A data-quality assertion evaluated against a model or source.
The proposed transition from current state to desired state.
Execution of an approved plan.
A logical data state. For Iceberg/Nessie deployments this should normally map directly to a Nessie reference.
Dependency graph between models and sources.
Higher-level graph containing workflows, tasks, sources, transformations and outputs.
Phlo Transform must support multiple transform roots natively.
phlo.toml
transforms/
├── shared/
│ ├── dimensions/
│ │ └── date.sql
│ └── reference/
│ └── sites.sql
│
workflows/
├── assay_ingest/
│ ├── workflow.toml
│ ├── tasks/
│ └── transforms/
│ ├── staging/
│ │ └── raw_results.sql
│ └── marts/
│ └── assay_results.sql
│
├── manufacturing/
│ ├── workflow.toml
│ └── transforms/
│ ├── batches.sql
│ └── deviations.sql
│
└── analytics/
└── transforms/
└── monthly_summary.sql
All of these belong to one transformation workspace.
Default transform-root conventions:
/transforms/**
/workflows/*/transforms/**
Additional paths may be configured:
[transforms]
include = [
"domains/*/transforms/**"
]
exclude = [
"**/target/**",
"**/.phlo/**"
]The default behaviour should be sufficient for ordinary projects.
A transform root may optionally contain transform.toml:
namespace = "assay_ingest"
schema = "assay"
materialized = "view"Where configuration is absent, the namespace is derived by convention. For example:
workflows/assay_ingest/transforms/
becomes namespace assay_ingest.
Filesystem location and model identity must be related but distinct.
Canonical internal URI:
model://assay_ingest/staging/raw_results
User-facing notation:
assay_ingest.staging.raw_results
Physical location:
workflows/assay_ingest/transforms/staging/raw_results.sql
A model may optionally pin its logical identity:
-- @id assay_ingest.raw_resultsThis allows files to move without losing historical identity and is useful for run history, incremental state, lineage, cached results, renames, audit trails and UI references.
If no explicit ID is provided, the compiler derives one deterministically.
Given:
workflows/assay_ingest/transforms/staging/raw_results.sql
The default identity may be:
assay_ingest.staging.raw_results
Given:
transforms/shared/dimensions/date.sql
The identity may be:
shared.dimensions.date
The preferred syntax is ordinary SQL:
select *
from assay_ingest.staging.raw_resultsRelations are resolved deterministically:
- exact fully-qualified workspace model;
- current namespace;
- current transform root;
- globally unique workspace model;
- declared external source;
- catalogue relation;
- compilation failure.
Any ambiguity produces an error.
error: ambiguous relation `results`
Candidates:
assay_ingest.results
manufacturing.results
Use a qualified model name.
For convenience and migration, restricted templating may support:
{{ ref("assay_ingest.results") }}and:
{{ source("lims", "samples") }}This is not the preferred native model. The compiler must not depend on ref() to construct its graph.
External relations should normally be inferred.
select *
from raw_lims.samplesIf no workspace model produces raw_lims.samples, the compiler checks the target catalogue. If the relation exists it is registered as an external source.
source://raw_lims/samples
Users may optionally enrich sources:
[source."raw_lims.samples"]
owner = "Analytical Development"
freshness = "24h"Source declarations are not mandatory merely to use the source.
Configuration precedence is limited to:
workspace defaults
↓
transform root
↓
folder
↓
model
↓
explicit CLI override
No additional hidden precedence layers.
Example:
[transform]
default_materialization = "view"
default_catalog = "phlo"
default_schema = "analytics"
[transform.execution]
max_concurrency = 8
[transform.state]
enabled = true
[transform.discovery]
include = [
"transforms/**",
"workflows/*/transforms/**"
]transforms/
└── assay/
├── transform.toml
├── staging/
└── marts/
materialized = "view"
[folder.marts]
materialized = "table"Nested folder configuration may be supported. Effective configuration must always be inspectable:
phlo-transform inspect assay.results --configExceptional model behaviour can be expressed using lightweight header directives:
-- @table
-- @key experiment_id
-- @not-null sample_id,result
-- @owner analytical-development
-- @tags assay,gold
select ...Equivalent long form:
-- @materialized tableDirectives should remain intentionally small and parsed independently of SQL syntax.
Model directives should not become a programming language. They should express declarative characteristics such as ID, materialisation, key, incremental strategy, tags, owner, nullability and contract behaviour. Anything requiring arbitrary computation belongs elsewhere.
Compilation pipeline:
discover files
↓
parse metadata
↓
parse SQL AST
↓
resolve relations
↓
construct graph
↓
load catalogue metadata
↓
resolve types
↓
derive column lineage
↓
validate contracts
↓
calculate model versions
↓
calculate execution plan
No warehouse mutation occurs during compilation.
The compiler must use a real SQL parser. sqlparser-rs is the preferred starting point.
Initial dialect focus:
- Trino;
- DuckDB;
- PostgreSQL.
An abstraction should exist for dialect-specific syntax but remain significantly simpler than dbt's adapter macro architecture.
Raw parser output should be converted into Phlo's own semantic representation:
Parsed SQL AST
↓
Resolved AST
↓
Typed AST
↓
Logical Model
For example:
select
r.sample_id,
r.result / s.volume as concentration
from assay.results r
join assay.samples s using (sample_id)produces a semantic representation equivalent to:
Projection
├── sample_id
│ └── assay.results.sample_id
│
└── concentration
└── Divide
├── assay.results.result
└── assay.samples.volume
The graph must know more than model-to-model edges.
assay.results.result
↓
assay.summary.mean_result
This supports model lineage, column lineage, impact analysis, schema-change detection, rename analysis and agent tooling.
Where sufficient metadata exists, compilation should detect:
- missing models;
- missing columns;
- ambiguous columns;
- invalid joins;
- incompatible types;
- invalid casts;
- circular dependencies;
- invalid model directives;
- incompatible materialisation configuration;
- contract violations;
- unresolved sources.
Dynamic SQL may require deferred runtime validation. The compiler must represent uncertainty rather than pretend certainty.
Each graph node may represent a model, source, test or workflow transform group. Model dependencies are derived primarily from SQL relations.
raw.results
↓
assay.clean_results
↓
assay.assay_results
↓
analytics.summary
Topological scheduling follows from this graph. Cycles are compilation errors.
Initial materialisations:
-- @viewCandidate default for lightweight staging models.
-- @tableFully materialised relation.
-- @ephemeralThe model is never materialised. References to it are inlined into dependents as a derived-table subquery at compile time — including nested ephemeral chains — and the model is excluded from the execution plan. Ephemeral models must be single-statement selects.
-- @incremental key=experiment_idOnly changed data is applied.
Users specify intent rather than warehouse-specific implementation.
Supported strategies:
append
key
partition
time-window
full
Examples:
-- @incremental append-- @incremental key=experiment_id-- @incremental partition=run_date-- @incremental window=updated_atThe adapter determines appropriate execution SQL.
A declaration such as:
-- @key experiment_idmay contribute to schema contract, uniqueness checks, non-null checks, merge strategy, record identity and data diffing. The same information should not need to be declared repeatedly.
The planner must identify dangerous state changes:
assay.results
incremental key changed:
old: sample_id
new: experiment_id
Action required:
FULL REBUILD
No silent incremental continuation should occur.
Testing is split conceptually into structural contracts and runtime assertions.
Structural contracts include column existence, expected type, nullability, key definitions and schema compatibility.
Runtime assertions include uniqueness, referential integrity, range checks and custom SQL conditions.
-- @key experiment_id
-- @not-null sample_id,resultOptional stricter contract:
[model.assay_results.contract]
enforced = true
[model.assay_results.columns.experiment_id]
type = "VARCHAR"
nullable = false
[model.assay_results.columns.result]
type = "DOUBLE"
nullable = falseColumn renames are declared alongside the contract so diffs can classify a removed+added pair as a named rename — a breaking change for consumers still selecting the old name — rather than an unexplained removal:
[model.assay_results.renames]
result = "legacy_result"Explicit contract files remain optional.
@key experiment_id should imply at least:
NOT NULL experiment_id
UNIQUE experiment_id
Where a relationship is declared, referential integrity testing should be generated automatically.
Custom tests may be ordinary SQL returning violating rows:
tests/
└── assay_results_positive.sql
select *
from assay.assay_results
where result < 0Zero rows means pass. This avoids inventing a complex test DSL.
Optional source metadata:
[source."lims.samples"]
freshness = "6h"
timestamp_column = "loaded_at"Planner output:
lims.samples
age: 7h 13m
required: <6h
status: STALE
Policy determines whether stale data warns, blocks or requires explicit override.
Each logical model produces a model-version fingerprint conceptually based on:
hash(
canonical SQL AST
+ effective configuration
+ compiler version
+ dependency versions
+ source version metadata
+ relevant target semantics
)
Example:
assay.assay_results@b74a02e
The engine distinguishes desired model version from currently materialised model version. Execution reconciles the two.
Phlo Transform is therefore closer conceptually to a build system than a SQL script runner.
If desired and materialised versions match, skip execution. If model code changes, build. If an upstream dependency changes materially, build. If only irrelevant metadata changes, skip where this can be determined safely.
For Iceberg sources, state may include:
- snapshot ID;
- schema ID;
- branch;
- partition metadata.
This makes source change detection substantially more precise than timestamp-only heuristics.
Successful immutable model versions may be cached. Where a compatible desired model version already exists, it may be reused or promoted instead of recalculated.
Cache reuse must account for environment and source-state compatibility. The model version hash is catalog-independent — the catalog is an environment binding, not model content — so logically identical content hashes identically in every environment and a version match across environments is possible at all. A version-hash match in another environment authorises reuse only when the record names the same content slot (schema.table — the catalogs may differ, which is exactly the Nessie branch-inheritance case), was produced by the same adapter, and carries a strong output identity (the adapter's post-write output_identity — e.g. the Iceberg snapshot id) that still matches what this environment's target relation reports — the hash says what should exist, not where it was written, which engine wrote it, or that it is still there. output_identity must prove physical contents: a schema fingerprint or row count describes shape, not bytes, so adapters that cannot prove identity (non-Iceberg relations, DuckDB) report none and never authorise reuse. Same-version records that fail any check produce a cache_miss build reason; a current-environment record produced by a different adapter or predating adapter tracking produces adapter_change, and one whose recorded output identity no longer matches the physical relation produces output_drift.
A cached plan action carries the matched record's provenance — source environment, source target, output identity, producing run and materialisation timestamp — and execution re-verifies the live identity before adopting it: a target that no longer reports the recorded identity falls back to a full build. Adoption persists an environment-local materialisation record that claims the source run id and timestamp verbatim — reusing an output produces no new materialisation — and copies the source environment's time-window watermark, since identical content carries the same frontier. The distinction is:
skip— this environment's own record already vouches for the desired version at this target;cached— another environment's record plus a live identity proof let this environment adopt the output without executing model SQL.
Materialisation records are ordered writes, not last-writer-wins: model_versions and seed_loads only apply records that are not older than what is stored, and watermarks order by run generation, so two runs completing out of order cannot regress shared state.
Planning is a core command:
phlo-transform planEvery decided model carries at least one structured reason; the human output prints each reason's detail under the model:
Plan: 4f6c…
Adapter: trino
Environment: feature/new-assay
Selection: assay.results+ — excluding assay.legacy_raw
Models (4) — 3 build, 1 skip, 0 cached
SKIP assay.reference [view] iceberg.assay.reference
SQL, config, contract and inputs unchanged
BUILD assay.clean_results [table] iceberg.assay.clean_results
SQL semantics changed
source raw.lims changed (snap:81f2… → snap:9c40…)
BUILD assay.assay_results [table] iceberg.assay.assay_results
upstream version changed: assay.clean_results
upstream assay.clean_results will rebuild in this plan
strategy: key
BUILD analytics.monthly_summary [table] iceberg.analytics.monthly_summary
selected by `assay.results+`
upstream assay.assay_results will rebuild in this plan
Tests (18)
…
Reason kinds are stable machine-readable codes (sql_semantic_change,
config_change, contract_change, dependency_change, upstream_rebuild,
source_change, target_change, compiler_semantics_change,
incremental_change, schema_change, missing_relation, unknown_state,
cache_reuse, cache_miss, adapter_change, output_drift, forced,
unchanged, selected_dependency, selection_expansion,
state_unavailable, git_change, resumed_run). --json exposes them with an
optional subject (the dependency or source the reason is about), plus the
resolved selection (terms, matched, expanded, required, exclude) and each
model's membership (selected / expanded / dependency).
phlo-transform applyIdeally, apply executes a previously calculated plan:
phlo-transform apply <plan-id>This guarantees execution matches the inspected plan unless source state invalidates it.
A plan becomes stale when relevant conditions change, including source snapshot, branch state, model files, dependencies or catalogue schema. Apply should reject stale plans unless explicitly replanned.
All planning commands share one selector engine (see §82). Examples:
phlo-transform plan assay.results # one model + its dependencies
phlo-transform plan assay.results+ # …plus transitive dependents
phlo-transform plan +assay.results # explicit upstream expansion
phlo-transform plan 'assay.*' # namespace glob
phlo-transform plan source:lims+ # models reading lims + dependents
phlo-transform plan --tag qc # intersect the selection by tag
phlo-transform plan --changed # desired version ≠ recorded state
phlo-transform plan --select assay.* --exclude assay.legacy_raw
phlo-transform plan --downstream assay.resultsPositional selector terms and --select are equivalent. --exclude is
absolute: an excluded model is never pulled back in by + expansion or by
dependency closure — a selected model that depends on it plans against the
existing materialisation and the plan records a warning. If the excluded
model was never materialised the plan is rejected outright (excluding an
ephemeral dependency is always fine — its SQL is inlined, so no relation is
needed).
For the primary Phlo environment, Nessie references should represent transformation environments directly.
Examples:
main
feature/assay-normalisation
ci/pr-184
release/2026-09
Avoid building a separate environment abstraction when Nessie already provides one.
A Git branch may map automatically or explicitly to a Nessie reference. Explicit form:
phlo-transform --ref feature/new-assay planReferences are managed explicitly — ref list, ref show, ref create --from <base> and ref delete — and no command creates or deletes a branch as a side effect, except explicit candidate provisioning on apply/run --ref and --cleanup on promote. plan/test --ref resolve the same environment read-only — they compute the physical catalog and compile against it without creating anything. ref delete main is refused: main is the default base, not a scratch branch. Provisioning is recorded per candidate as an immutable environment evidence record in the state store — the portable authority promotion audits — and exported to environment_<sanitised ref>_<hash>.json (plus the single-slot environment.json); the evidence is deleted with the branch. Each record carries created_from — the reference and commit the candidate was provably created from, recorded at ref create or first provisioning and preserved across re-provisioning — and the physical catalog plus a catalog_status (created / unverified / unmanaged). A pre-existing branch Phlo did not create has unrecorded provenance; promotion refuses it rather than redefine its base as the current target head.
Candidate catalogs are named phlo_<sanitised-ref>_<hash> — the readable ref plus 8 hex of its SHA-256 — so punctuation-equivalent refs (ci/pr-1, ci_pr_1) can never collide on one physical catalog. Because a catalog's bound Nessie ref cannot be read back over SQL and the generated name is a public convention anyone can mint, an existing catalog is never adopted on name alone: unverified catalogs are accepted only when recorded environment evidence binds that catalog to this ref (a record that vetted the catalog — recorded together with the catalog_owned_by_phlo ownership flag, so a pre-flag unverified record that may have been adopted on the name alone is not evidence — or an explicit ref create --catalog pin), and a catalog another candidate's evidence claims is refused outright. A rejected catalog rolls back a just-created candidate branch so a failed provisioning attempt leaves nothing behind.
Production-oriented execution follows:
WRITE
↓
AUDIT
↓
PUBLISH
Typical sequence:
create/update branch state
↓
execute models
↓
run contracts
↓
run tests
↓
run data diffs
↓
apply policy
↓
promote Nessie reference
Transforms should not write directly to canonical production state by default.
Example:
phlo-transform promote feature/new-assay --to main
phlo-transform promote --from feature/new-assay --to mainPromotion is authorised by named gates, reported identically in human and JSON output: run (latest candidate run passed and validated the exact commit being promoted — a passed run is bound to the candidate head after its writes land, and a candidate that advanced since its run, or a run recorded before commit binding, fails), tests (no failed tests), blocked (no blocked/cancelled model, seed or test work), schema (a fresh audited diff inspected this pair at these commits and found no unwaived breaking changes — physical breaks come from the audited diff; contract breaks are computed live from the workspace's desired contracts against the target's recorded contracts, so a post-diff contract edit cannot slip past — and absent or stale evidence fails closed, never reading "no evidence" as "no changes"), data_diff (when --require-diff is set: a passing --full audited diff bound to this candidate→target pair at the commits being promoted — the evidence records both refs' resolved heads and is rejected when they no longer match — still fresh per recorded versions, and never a self-comparison; a single-model diff.json is never promotion evidence), base (the target still equals the commit the evidence was established against — the hash-bound audit's recorded base, else the candidate's created_from provenance; unknown provenance fails rather than redefining the base as the current head, and the merge asserts the evaluated target hash) and conflicts (the merge check is clean). A candidate that advanced between gate evaluation and merge is refused rather than promoted unaudited. --check evaluates gates without merging; a passing promotion merges and persists a PromotionRecord (refs, hashes, plan/run ids, gate results, timestamp) in the state store.
Preconditions may include successful plan, successful execution, required tests passing, no blocking schema changes, no stale state and optional approval.
Because published table state is versioned, rollback is a first-class operation:
phlo-transform rollback assay.results --to <version>or through environment-level Nessie rollback semantics.
Schema changes must be understood structurally.
assay.results
Schema changes
+ dilution_factor DOUBLE
~ result FLOAT -> DOUBLE
- legacy_result VARCHAR
Classification:
SAFE
+ nullable column
REVIEW
numeric widening
BREAKING
removed column
Workspace policy may define behaviour:
[schema]
allow_nullable_additions = true
allow_numeric_widening = true
breaking_changes = "error"Removing assay.results.legacy_result may produce:
Affected downstream models:
analytics.legacy_report
direct reference at line 18
qc.monthly_extract
SELECT * dependency
This must be available during planning before data is modified.
Native command:
phlo-transform diff assay.results
phlo-transform diff --from feature/new-assay --to mainWith no model argument, diff compares two Nessie references: every dataset known to the workspace or recorded in state is classified added/removed/changed/unchanged/absent, schema and nullability changes are listed per model, row counts come from the catalogs, and --full runs keyed value diffs on changed models. main's records include the default (unlabeled) environment, so state from a run with no --ref still counts. When Nessie is configured the report records both refs' resolved commit hashes, binding the evidence to exact branch heads. The report is persisted as branch-diff evidence in the state store (exported to branch_diff.json) — the audit promotion consumes.
Example:
assay.results
Rows
main 128,231
candidate 128,237
delta +6
Records
added 12
removed 6
modified 31
Changed values
concentration 23
status 8
sample_id 0
Possible comparison strategies:
key-based
partition-based
aggregate
sampled
full
Large-table defaults should avoid expensive full comparison.
phlo-transform diff assay.results --fullA workflow may require:
[model.assay_results.diff]
max_removed_rows = 0
max_changed_fraction = 0.05This may become a promotion quality gate. Row thresholds require keyed coverage to measure; on a keyless model (or under --partition) they fail rather than pass on unmeasured zeros.
Compilation produces one canonical LineageGraph — model, dataset, column
and test nodes joined by input, output, derives, contains and tests
edges. Model outputs, sources and seeds are all dataset:// nodes, so lineage
is expressed between datasets rather than being tied to transform models;
future ingestion systems contribute nodes and edges to the same graph.
Every input connects identically: dataset ──input──▶ model covers sources,
seeds and upstream model outputs alike (model → model and dataset → dataset edges exist only as one-hop rollups). Tests are consumers — dataset ──tests──▶ test — so impact traversal reaches them naturally.
Every report and export reads this one structure — lineage, impact,
artifacts and the OpenLineage exporter never re-derive their own:
Compilation ─▶ LineageGraph ─▶ OpenLineageExporter
Column derives edges carry optional metadata: directness
(direct/indirect), transformation (identity, transformation,
aggregation, join, filter, group_by, sort, window, conditional),
confidence (exact for AST-proven lineage, unknown when analysis is
incomplete; inferred/declared/runtime reserved for other producers) and
the responsible SQL expression. Join keys, filters, grouping and sort keys are
recorded as indirect inputs rather than dropped.
The graph is indexed and deterministic; document() serialises it and
document_for/subgraph scope it to a selection. See
docs/lineage.md.
phlo-transform lineage assay.assay_resultslims.results
↓
assay.raw_results
↓
assay.clean_results
↓
assay.assay_results
↓
analytics.monthly_summary
lineage --format graph prints the canonical document;
lineage --format openlineage exports the OpenLineage design-time document —
a bare JSON array of spec-valid JobEvents and DatasetEvents, each with
eventTime/producer/schemaURL, usable directly as a batch-endpoint
payload. Both accept a model target or selector terms for scoping.
lineage --diff <git-ref> is the semantic complement to the data-side
branch_diff: the workspace subtree at merge-base(ref, HEAD) is
materialised read-only (ls-tree/cat-file, no worktree or checkout
mutation), compiled with the same options, and the two graphs are compared
— nodes and edges added, removed or field-changed, plus the consumers each
removed node orphans and the downstream each removed or moved edge affects.
Edges compare as multisets: parallel edges between the same nodes keep
their metadata distinct. lineage --diff <base> <candidate> compares two
exact refs instead. The report persists as lineage-diff evidence in the
state store — exported to lineage_diff.json — with the
resolved base commit, the candidate's git head and worktree state, and the
Nessie branch-pair binding when --ref/--from resolves — promote
audits that provenance and reports the delta as current, advisory, or
stale with the reason rather than treating a moved report as evidence.
phlo-transform lineage assay.assay_results.resultlims.raw_results.signal
↓
assay.raw_results.signal
↓
assay.clean_results.corrected_signal
↓
assay.assay_results.result
The report separates direct inputs from indirect ones (join keys, filters,
grouping/sort keys) and prints the column's confidence when it is not
exact.
phlo-transform impact assay.results.result
phlo-transform impact external.samples.volumeReturns downstream columns, models, workflows, tests and registered published consumers. Source and seed columns are valid targets — the walk traverses the canonical graph, so a raw input column reports every downstream column and model it feeds.
Transforms may belong to a workflow by physical ownership:
workflows/
└── assay_ingest/
├── workflow.toml
├── tasks/
└── transforms/
The workflow engine should not need to enumerate every transform model. A transform namespace may expand internally into its transformation DAG.
High-level workflow:
extract
↓
validate
↓
transform
↓
publish
Transform expands to:
raw_results
↓
clean_results
↓
assay_results
Combined graph:
extract
↓
validate
↓
raw_results
↓
clean_results
↓
assay_results
↓
publish
This creates one coherent workflow/data-lineage representation.
A model may reference another workflow's model:
select *
from manufacturing.batchesinside workflows/analytics/transforms/ creates:
manufacturing.batches
↓
analytics.monthly_summary
This must not require publishing packages between separate transformation projects.
Cross-workflow references may optionally trigger policy:
[dependencies]
cross_workflow = "warn"A future visibility model may distinguish workflow-private, workspace-public and external-public models.
Execution is graph-based and requires:
- topological scheduling;
- bounded concurrency;
- dependency-aware failure handling;
- retries for transient failures;
- cancellation;
- structured events;
- deterministic status;
- per-model timing;
- target-specific transactions where applicable.
Tokio is the expected async runtime.
A model execution may be:
pending
ready
running
passed
failed
skipped
blocked
cancelled
cached
blocked indicates an upstream dependency prevented execution — the model's own SQL never ran, and it is not reported as a model failure. skipped and cached satisfy dependents; failed, blocked and cancelled do not.
Independent branches may continue when safe.
A → B → C
D → E
If B fails:
A PASS
B FAIL
C BLOCKED
D PASS
E PASS
With --fail-fast, the first unrecoverable failure instead stops new scheduling: dependents of the failure are blocked, unrelated not-started work is cancelled, and in-flight tasks are aborted. Each attempt runs through a tracked adapter view that reports its in-flight query ids, so abort and per-attempt --model-timeout cancel the underlying warehouse query where the adapter supports it (Trino: DELETE /v1/query/{id}); adapters that cannot report in-flight queries degrade to dropping the attempt's future.
Retry applies only to recognised transient execution failures by default. Compilation errors and deterministic test failures must not be retried automatically.
Every failure carries a stable machine-readable category (adapter, sql, test, timeout, cancelled, dependency, state, internal) plus the adapter error code, attempt number, timestamp and a retryable flag. --retries N allows extra attempts with bounded exponential backoff; only adapter failures the adapter marked retryable are retried — SQL-semantic errors are never retried, and test/timeout/dependency/cancellation failures are scheduler or assertion outcomes, not transient faults. Every attempt is recorded.
Execution progress is persisted incrementally: the run row and stored plan are written before scheduling, each node record lands as it transitions, and every failed attempt is persisted before its retry backoff begins. A state-store write failure is fatal to the run — progress that cannot be recorded cannot be trusted for resume. A killed process therefore leaves an accurate partial record rather than a misleadingly successful one.
Two continuations rebuild from that state:
run --resume <run-id>— continue an interrupted run (stillrunning, orcancelled) under the same run id. Verifiedpassedwork is reused (a model's earlierpassedcounts only when its desired version still matches and its target relation still exists; a seed's only when its content hash matches and its target exists); every other node is re-planned through the normal planner against current state — storedskip/cacheddecisions,full_rebuildflags and watermarks are never trusted once the workspace moved, so an incremental strategy/key change or a schema change classified as full-rebuild-required rebuilds fully, and time-window models read the current watermark. A finished run is refused:failedredirects to--retry-failed.run --retry-failed <run-id>— start a new run (linked bycontinued_from) over the failed/blocked/cancelled models of a finished run, the dependencies they still need, and any tests that failed; tests over rebuilt models re-verify too.
Only passed materialisations update materialised-version state; a model version is never recorded as successful before execution genuinely completes.
Adapters expose a compact Rust trait conceptually similar to:
trait Adapter {
async fn introspect_relation(...);
async fn execute(...);
async fn create_table(...);
async fn create_view(...);
async fn apply_incremental(...);
async fn drop_relation(...);
async fn get_source_state(...);
}Avoid executable macro dispatch.
- v1: Trino;
- v1.x: DuckDB;
- v2: PostgreSQL.
Additional adapters are justified only by concrete use cases.
Primary responsibilities:
- SQL execution;
- metadata introspection;
- Iceberg relation management;
- CTAS;
- view creation;
- incremental write strategy;
- query status;
- cancellation;
- errors;
- statistics where available.
The primary adapter should understand Iceberg concepts explicitly, including snapshot ID, schema ID, partition specification, table properties and commit history. These feed planning and state comparison.
Nessie operations should be separate from SQL execution.
Responsibilities:
- branch creation;
- branch lookup;
- hash resolution;
- merge;
- conflict detection;
- branch deletion;
- rollback;
- environment comparison.
Every run should emit structured artifacts:
.phlo/
└── transform/
├── manifest.json
├── graph.json
├── plan.json
├── run.json
├── lineage.json
├── openlineage.json
└── state/
These files are interfaces, not incidental logs.
manifest.json includes workspace, transform roots, models, sources, IDs, paths, effective configuration, materialisations, dependencies, columns, contracts, tests and model hashes.
graph.json includes nodes, edges, edge types, column edges, workflow relationships and external-source relationships.
lineage.json holds the canonical lineage document — every model, dataset, column and test node with its edges and column-level metadata — and openlineage.json the same graph exported as an OpenLineage design-time document (a JSON array of spec-valid JobEvents and DatasetEvents) for tools such as OpenMetadata and DataHub.
plan.json includes plan ID, current environment state, desired state, models selected, the resolved selection (terms, excludes, matched/expanded/required provenance), structured per-model reasons with stable kind codes, proposed physical operations, schema impacts, tests, data-diff requirements and promotion constraints.
run.json includes run ID, plan ID, continued_from, environment, the candidate reference hash the run validated (reference_hash, bound post-run), timestamps, model/seed/test results, compiled SQL hashes, query IDs, per-status counts, model versions, source versions, timings, per-attempt records and structured failures (category, adapter code, retryable flag).
Primary interface:
phlo-transform
Core commands:
phlo-transform check
phlo-transform plan
phlo-transform apply
phlo-transform run
phlo-transform test
phlo-transform inspect
phlo-transform explain
phlo-transform lineage
phlo-transform impact
phlo-transform diff
phlo-transform list
phlo-transform promote
phlo-transform rollback
phlo-transform clean
phlo-transform state runs|show|model|promotionsThe state backend is selected by --state/PHLO_STATE_URL: a postgres:///postgresql:// URL selects the shared PostgreSQL store; anything else is a SQLite file path (default .phlo/transform/state.db).
Compile and statically validate without planning execution:
phlo-transform checkOutput includes SQL parse errors, unresolved relations, ambiguous references, circular dependencies, type mismatches and contract errors.
It should be fast enough for frequent local use.
Convenience command, broadly plan + apply, intended for development environments. Production use should favour explicit plan/apply.
Execution flags apply to run/apply alike:
phlo-transform run --jobs 4 # bounded concurrency
phlo-transform run --retries 2 # transient-failure retries
phlo-transform run --fail-fast # stop scheduling on first failure
phlo-transform run --model-timeout 30m # per-attempt timeout
phlo-transform run --resume <run-id> # continue an interrupted run
phlo-transform run --retry-failed <run-id> # new run over the failed portion--resume/--retry-failed take a full run id or a unique prefix; selection flags do not apply to them — the prior run defines the work.
phlo-transform inspect assay.resultsExample:
Model: assay.results
ID: model://assay/results
Path: workflows/assay/transforms/results.sql
Materialisation: incremental
Key: experiment_id
Depends on:
lims.results
shared.samples
Used by:
analytics.monthly_summary
State:
desired: b74a02e
current: a621ee3
status: changed
explain is the per-model companion to plan: identity, dependencies,
recorded state, and the current build decision with the same structured
reasons plan reports (the state diff needs no adapter; when one is
configured the relation-existence check makes the decision identical to
plan):
phlo-transform explain assay.resultsModel: assay.results
…
Version: b74a02e…
Recorded: a621ee3… in dev (materialised 2026-02-10T08:41:07Z)
Decision: build
Reasons:
upstream version changed: assay.clean_results
upstream assay.clean_results will rebuild in this plan
Every relevant command should support --json.
No semantic information should exist only in formatted CLI output.
One selector language is shared by plan, apply, run, test,
lineage, impact and list. Per term:
term := "+"? body "+"?
body := "tag:" value model carries the tag
| "namespace:" value model's namespace (first name segment)
| "source:" value model reads a matching source
| "changed" desired version differs from recorded
state — or, under `--since`, from a
Git ref
| "all" | "*" every model
| pattern name, `model://` URI, `prefix.*` glob,
or a unique name suffix
A leading + adds every transitive dependency of the matched models; a
trailing + adds every transitive dependent; +model+ does both. Include
terms union; --tag/--workflow intersect; --exclude terms subtract last
and are absolute (excluded models are never re-added by expansion or
dependency closure). Selection is deterministic — topological order where
the graph allows — and errors are explicit: empty, invalid, ambiguous and
unmatched selectors all fail with named candidates or suggestions rather
than silently widening.
CLI mapping:
phlo-transform plan assay.results 'assay.*' # positional terms
--select tag:qc # same grammar via flag
--exclude assay.legacy_raw # subtract (applied last)
--tag qc --workflow assay_ingest # intersect filters
--changed # shorthand for `changed`
--since main # `changed` resolves via Git
--upstream / --downstream # expand the filtered base
--force # rebuild regardless of stateThe changed term has two providers answering different questions:
- State-derived (default): the desired model version differs from the version recorded as materialised for the target environment. Because a moved upstream version changes dependents' versions, this set naturally includes downstream models. Ephemeral models are never reported changed — they are never materialised; their edits surface through dependents' dependency versions.
- Git-derived (
--since <ref>): semantic project inputs changed relative to a Git ref — the directly changed set only. Downstream impact comes from selector expansion (changed+), not from the diff.
phlo-transform plan --since main # shorthand for --select changed --since main
phlo-transform plan --since main --select changed+
phlo-transform run --since origin/main
phlo-transform impact --since main # blast radius of the diff--since compares merge-base(<ref>, HEAD) against the working tree —
the fork point, so a moving main doesn't inflate the diff — and counts
staged, unstaged and untracked files. It requires a changed term
somewhere in the selector set (implied when no include terms are given);
--since <ref> <model> without changed is an error rather than a
silently ignored flag.
The Git provider maps each changed workspace path once — model files
(semantic comparison: canonical SQL plus directives, so comment- and
formatting-only edits don't count), seeds/*.csv (consumers of the seed's
source relation; unused seeds are still reported), tests/*.sql (the
test's target models, so test --since covers them), transform.toml
(every model under its directory) and phlo.toml (narrowed to changed
sections; [transform]/[dependencies]/defaults widen to all models).
Added, modified, renamed, deleted and untracked files all count; a deleted
or renamed-away model marks its dependents, since they now read a source
where a model used to be. When a path cannot be proven narrow, the
provider widens rather than guesses.
Direct changes carry git_change selection provenance into the plan —
distinct from rebuild reasons, which stay state/version-based:
BUILD assay.raw selected because transforms/assay/raw.sql modified since main
SQL semantics changed
BUILD assay.results selected by `changed+`
upstream assay.raw will rebuild
Plan JSON exposes plan.git: the requested ref, resolved merge-base and
HEAD, every changed workspace path with its status, the directly changed
models with causes, changed seeds and consumers, changed tests, deleted
model identities, and paths outside the workspace.
Phlo Transform must be explicitly designed for machine interaction.
Examples:
phlo-transform inspect assay.results --json
phlo-transform lineage assay.results --json
phlo-transform impact assay.results.result --json
phlo-transform plan --jsonAgents should query compiler truth directly.
Agents should be able to answer:
- what does this model depend on?
- where does this column originate?
- what breaks if this column is removed?
- which models changed?
- why will this model rebuild?
- what will this PR change in production?
- which models failed previously?
- what is the inferred schema?
- which workflows consume this dataset?
- what are the current quality gates?
without manually parsing source files.
phlo-transform daemonThe daemon maintains an in-memory workspace representation containing parsed ASTs, typed ASTs, dependency graph, catalogue schemas, lineage, hashes, filesystem state and compiled plans, published as an immutable Arc<Compilation> snapshot so readers never observe partial updates.
It serves a versioned local HTTP API (see docs/daemon.md) with two planes:
- Reads (
GET /v1/...): check, models, inspect, lineage (document, model, column), impact (model, column, selection), graph, plan, branch/lineage diffs, and state inspection (runs, run detail, run failures, materialisations, promotions) — the same report DTOs as--jsonoutput. - Operations (
POST /v1/operations):run,resume,retry_failed,test,promote,reloadwith a stablequeued → running → succeeded|failed|cancelledlifecycle, live progress read back from the state store, cooperative cancellation viaCancelHandle, request-bound idempotent replay (idempotency_key/Idempotency-Keyheader; keys are global to the endpoint — the same key under different params or a different kind isAPI016), and a single-mutating-operation gate (API008). The idempotency reservation is journaled to.phlo/transform/operations.jsonlbefore the operation is acknowledged, so a restart can never execute a keyed submission twice; an operation in flight at shutdown is restored asfailed(interrupted).
Environment-targeted operations run the same provisioning the CLI applies for --ref: with a catalog-facing Nessie URI configured, run/resume/retry_failed ensure the candidate branch and its branch-scoped catalog, recompile the workspace retargeted at it, and bind a passed run to the environment's post-run Nessie head — main is never written. When Nessie is configured but branch isolation cannot be provisioned (no catalog-facing URI or adapter), a non-base environment run fails with API007 rather than recording falsely-labelled evidence; without any Nessie client the environment is an honest state label. A continuation targets the stored run's environment, not a requested one.
Operations and reads share one environment resolution path (EnvironmentContext::resolve). Mutating work resolves in Ensure mode — provision, then compile retargeted. GET /v1/plan?environment=&base= resolves in ReadOnly mode — the same physical catalog (override → recorded binding → generated phlo_<ref>_<hash> name) and a retargeted compile, but no branch created and no evidence written. The plan it previews names exactly the targets a run against that environment executes — including failure: when resolution lands on the generated catalog and the adapter cannot provision catalogs, ReadOnly fails API007 just as Ensure would, rather than preview a target no run reaches. An explicit or recorded catalog pin remains the escape hatch.
Promotion through the API uses the same audited evidence path as the CLI (phlo_transform_engine::audit): the gate reads the state store's evidence records — the branch-diff evidence (bound to both refs' resolved heads), the recorded provisioning environment (created_from provenance), the lineage-diff evidence's provenance (including the candidate's lineage fingerprint) — plus live contract diffs against the target's recorded contracts and the Nessie merge check; the *.json artifacts are exports of the same records. CLI and API cannot drift apart on what counts as evidence. An unbound run cannot promote a commit-bound candidate.
Errors are {"error": {"code", "message"}} with stable API0xx codes. daemon --token <secret> requires Authorization: Bearer on every endpoint except /status (API015).
Potential consumers include CLI, Phlo UI, editor extensions, workflow engine, AI agents, CI tooling and developer tools.
Communication may use local HTTP, Unix sockets, JSON-RPC or gRPC. Initial implementation should favour the simplest stable interface.
Filesystem changes should invalidate only affected compiler state:
assay/results.sql changed
↓
reparse results
↓
re-resolve its dependencies
↓
invalidate downstream typing/lineage
Do not recompile the entire workspace unnecessarily.
Target local loop:
edit SQL
↓
instant static feedback
↓
phlo-transform plan
↓
inspect changes
↓
apply
The compiler should be fast enough that check can reasonably run on save in moderate projects.
Errors must be typed, deterministic, location-aware, actionable and machine-readable.
error[T012]: unknown column `reslt`
--> workflows/assay/transforms/results.sql:14:9
14 | reslt
^^^^^
Did you mean:
result
Source:
assay.raw_results
Suggested categories:
PROJECT
CONFIG
PARSE
RESOLUTION
TYPE
GRAPH
CONTRACT
PLAN
ADAPTER
EXECUTION
TEST
DIFF
PROMOTION
STATE
Every error should have a stable error code.
Structured events should cover compilation, planning, model queueing, model execution, query IDs, test execution, retries, state comparison, promotion and errors.
Human logs and structured logs should be produced from the same event stream.
Run artifacts should provide enough information to reproduce and explain an execution, including code/model version, compiler version, effective configuration, dependency versions, source versions, plan, executed SQL hash, execution result, tests, environment and promotion result.
This positions Phlo Transform well for controlled scientific and regulated environments without making the core engine specific to one regulatory regime.
Given the same workspace, configuration, compiler version, source state and target semantics, planning must produce the same desired model versions and graph.
Nondeterministic configuration evaluation should be prohibited.
Templating support should be intentionally restricted. Possible supported constructs:
ref()
source()
var()
simple if
simple for
small built-in macro library
The compiler must avoid arbitrary Python execution.
Preferred:
select *
from assay.resultsSupported compatibility:
select *
from {{ ref("assay.results") }}Any feature achievable with ordinary SQL should prefer ordinary SQL.
Variables should be used sparingly:
[vars]
site = "oxford"where site = '{{ var("site") }}'Environment-specific relation names should ideally be handled by catalogue/environment resolution rather than variable interpolation.
Initial macro support should be a fixed built-in function set. Do not initially support arbitrary project-defined executable macros.
Future custom macros, if added, should ideally operate over SQL AST structures rather than arbitrary text.
The compiler should canonicalise SQL ASTs for hashing. Irrelevant formatting should not trigger rebuilds.
These should ideally hash equivalently:
select * from assay.resultsand:
SELECT
*
FROM assay.resultsComments unrelated to directives should similarly not necessarily invalidate materialisation state.
Longer term, state calculation may distinguish:
text change
AST change
semantic change
physical result-affecting change
Initial implementation may use canonical AST hashing.
Much documentation can be generated automatically.
For each model:
- description;
- path;
- owner;
- materialisation;
- inferred schema;
- dependencies;
- consumers;
- lineage;
- tests;
- run history.
Descriptions may optionally be supplied through comments or config. No separate documentation site is required initially.
Phlo UI can consume compiler artifacts or APIs to render workspace graphs, model details and plan review.
Model detail should include SQL, schema, lineage, history, tests, state and diffs.
Plan review should include models changing, schema impact, data impact, tests and promotion status.
Suggested workspace:
crates/
├── phlo-transform-core/
├── phlo-transform-project/
├── phlo-transform-parser/
├── phlo-transform-sql/
├── phlo-transform-graph/
├── phlo-transform-types/
├── phlo-transform-state/
├── phlo-transform-plan/
├── phlo-transform-executor/
├── phlo-transform-test/
├── phlo-transform-lineage/
├── phlo-transform-diff/
├── phlo-transform-adapter/
├── phlo-transform-trino/
├── phlo-transform-duckdb/
├── phlo-transform-nessie/
├── phlo-transform-artifacts/
├── phlo-transform-cli/
└── phlo-transform-daemon/
Do not over-fragment crates during early development. Initial implementation may consolidate several of these until interfaces stabilise.
Conceptually:
struct Workspace {
roots: Vec<TransformRoot>,
models: ModelRegistry,
sources: SourceRegistry,
graph: TransformGraph,
}struct Model {
id: ModelId,
namespace: Namespace,
path: PathBuf,
sql: ParsedSql,
resolved_sql: ResolvedSql,
config: ModelConfig,
schema: Option<ModelSchema>,
dependencies: Vec<RelationId>,
columns: Vec<Column>,
}struct ModelVersion {
model: ModelId,
version: ContentHash,
sql_hash: ContentHash,
config_hash: ContentHash,
dependencies: Vec<ModelVersionRef>,
sources: Vec<SourceVersionRef>,
}Use a DAG-capable Rust graph structure such as petgraph.
Potential edge types:
enum DependencyKind {
Model,
Source,
Column,
Workflow,
Test,
}Model-level and column-level graphs may be stored separately where operationally simpler.
If Jinja compatibility is retained, a constrained engine such as MiniJinja is suitable. Expose only allowed functions and constructs; do not expose arbitrary host-language functions by default.
Initial local state may use SQLite or DuckDB.
Requirements:
- model versions;
- run history;
- plan history;
- source states;
- execution metadata.
Phlo deployments may later use PostgreSQL. The storage interface should be abstracted.
Artifacts are immutable execution outputs. State storage is queryable operational history. Do not rely solely on JSON files for long-term operational state.
Typical CI:
checkout
↓
phlo-transform check
↓
phlo-transform plan --base main
↓
create CI Nessie branch
↓
apply
↓
tests
↓
diff
↓
report
CI artifacts should contain the complete plan and results.
Phlo could generate:
Transform changes
Models
4 changed
2 downstream rebuilt
Schema
+1 column
0 breaking changes
Data
+12 rows
-0 rows
31 records modified
Tests
27 passed
0 failed
For roughly 1,000 models, design toward:
cold check: <5 seconds
incremental check: <500 ms for common edits
These are design targets rather than hard MVP requirements.
Scheduling overhead should be negligible relative to warehouse queries. The semantic graph should remain comfortably resident in memory for normal enterprise workspaces.
The compiler itself should not require unrestricted database permissions.
Separate credentials where practical for read/introspection, development writes, branch writes and production promotion.
Sensitive credential values must never be embedded in model artifacts or logs.
[transform.execution]
max_concurrency = 8The scheduler should respect graph readiness, adapter limits, warehouse limits and optional resource pools.
Future:
[resources]
trino_heavy = 2
trino_normal = 8Later, models may specify:
-- @resource heavyThis can influence scheduling without exposing warehouse-specific knobs throughout model definitions. Not required for MVP.
Phlo Transform should exist as:
- Rust library;
- CLI;
- structured artifact interface;
- later daemon/service API.
The CLI should remain a thin consumer of core libraries.
Potential high-level Rust interface:
let workspace = Workspace::load(path)?;
let compilation = workspace.check().await?;
let plan = compilation.plan(target).await?;
let result = plan.apply().await?;The Phlo workflow engine should use this API directly rather than shelling out where practical.
Extension points should exist only where there is a known abstraction boundary.
Good extension points:
- adapters;
- state backends;
- catalogue providers;
- policy engines;
- event sinks.
Avoid arbitrary executable plugin systems initially.
Implementation details and acceptance criteria are broken out under docs/roadmap/.
The high-level phases are:
- Compiler spike;
- MVP build engine;
- typed compiler and lineage;
- state-aware execution;
- incremental models;
- Nessie and WAP;
- data diff;
- workflow integration;
- daemon and agent APIs.
The first genuinely usable release must support:
workflows/
├── assay/
│ └── transforms/
│ ├── raw.sql
│ └── results.sql
│
└── reporting/
└── transforms/
└── monthly.sql
results.sql:
select *
from assay.rawmonthly.sql:
select *
from assay.resultsWithout additional dependency configuration, the engine must derive:
assay.raw
↓
assay.results
↓
reporting.monthly
It must then:
- compile the workspace;
- detect syntax errors;
- detect unresolved relations;
- construct the DAG;
- create a plan;
- execute in dependency order;
- run tests;
- persist run state;
- skip unchanged models on subsequent runs;
- expose structured model metadata.
A mature release should allow a developer to run:
phlo-transform planand understand what changed, why, what will execute, what data may change, what schemas will change, which downstream models are affected, what tests will run and whether promotion is safe.
Then:
phlo-transform applyshould execute only required work, run appropriate tests, persist reproducible execution state, generate complete artifacts and leave production unaffected until promotion.
Then:
phlo-transform promoteshould atomically publish approved state where supported.
| dbt-style concept | Phlo Transform |
|---|---|
| project-per-model-tree | workspace with many roots |
mandatory ref() dependency model |
SQL relation resolution |
| mandatory source declarations | source inference |
| YAML-heavy metadata | inference + optional config |
| Jinja-heavy compilation | mostly real SQL |
| arbitrary macros | constrained functions |
| adapter macros | Rust adapter traits |
| multiple config precedence paths | fixed hierarchy |
| runtime-first validation | compile-first validation |
| state as optimisation | state as core execution primitive |
| environment abstraction | Nessie references where available |
| manual WAP patterns | native WAP |
| separate data/workflow lineage | unified graph |
| textual SQL understanding | typed semantic model |
Phlo Transform succeeds by being better specifically at:
- workspace-native transformations —
workflows/X/transformsworks naturally; - SQL-native dependencies — no
ref()required in ordinary cases; - minimal configuration — infer anything that can be inferred safely;
- typed compiler — understand models and columns structurally;
- column-level lineage — native compiler output;
- build-system semantics — desired state versus current state;
- plan/apply — changes are inspectable before execution;
- Nessie-native environments — versioned data and transformation state align;
- WAP — safe publication is the normal production execution model;
- native data diff — understand actual output changes, not only SQL changes;
- unified workflow and data lineage — transformations belong naturally to wider workflow execution;
- agent-native interfaces — AI systems query structured compiler state instead of reverse-engineering source trees.
SQL
│
┌─────▼─────┐
│ Compiler │
└─────┬─────┘
│
┌───────────┼───────────┐
▼ ▼ ▼
Typed DAG Lineage State
│ │ │
└───────────┼───────────┘
▼
Plan
│
┌──────┴──────┐
▼ ▼
Build Diff
│
▼
Audit
│
▼
Promote
The compiler is the core product. Execution, lineage, testing, planning, state, agents and workflow integration all derive from the same semantic representation.
Every new feature should be tested against this question:
Can the compiler infer this reliably instead of requiring the user to configure it?
If yes, infer it.
If no:
Can it be expressed declaratively with one small, predictable mechanism?
If yes, configure it once.
Only after both answers are no should a more complex abstraction be introduced.
That principle should keep Phlo Transform substantially smaller than dbt while allowing it to become more capable in the areas that matter to Phlo.