Skip to content

v0.1-release-notes.md

Latest commit

 

History

History
141 lines (116 loc) · 6.01 KB

v0.1-release-notes.md

File metadata and controls

141 lines (116 loc) · 6.01 KB
Raw

Phlo Transform v0.1.0

What it is

Phlo Transform is a workspace-native SQL transformation compiler and execution engine written in Rust. It parses ordinary SQL, resolves the workspace into one typed dependency graph without ref(), and treats planning, state, lineage, testing, diffing and audited promotion as first-class, compiler-driven concepts. It is intentionally not a dbt compatibility layer — it is a smaller, more predictable SQL build system.

Highlights

  • Plan before apply. Every plan explains what will run and why — per-model actions (build/skip/cached) with named reasons, before anything mutates.
  • Content-addressed state. Model versions hash semantics, not text: formatting and comments never trigger rebuilds; upstream changes cascade correctly downstream.
  • Executable cache reuse. cached is real: a Nessie candidate adopts a main materialisation with zero model SQL, verified live against the physical table identity (Iceberg snapshot), and falls back to a real build when evidence is stale.
  • Nessie-native environments and WAP. run --ref provisions an isolated candidate branch + catalog; promote merges only after audit gates (staleness, branch diff, breaking-schema, lineage diff) pass.
  • Column lineage and contracts. Canonical lineage graph, column-level impact analysis, OpenLineage export, schema contracts with breaking-change gates, declared renames.
  • dbt migration. translate --from dbt lowers dbt projects into native Phlo workspaces with an honest CLEAN/REVIEW/UNSUPPORTED classification and a written report — it does not pretend Jinja programs are SQL.
  • Machine API. phlo-transform daemon exposes the same compiler/engine truth over versioned local JSON (/v1/...) with tracked operations, idempotency keys and cooperative cancellation.

Core workflow

phlo-transform init
phlo-transform check
phlo-transform plan --adapter duckdb
phlo-transform run --adapter duckdb        # plan + apply + tests
phlo-transform test --adapter duckdb
phlo-transform explain <model>

--since <git-ref> plans and runs only what changed relative to a ref. --resume/--retry-failed continue interrupted runs; --retries N retries retryable adapter failures with bounded backoff.

Lakehouse / WAP

With Trino + Iceberg + Nessie configured (--trino-endpoint, --nessie-endpoint, optionally --nessie-catalog-uri when the warehouse reaches Nessie on a different address):

phlo-transform run --adapter trino --catalog <cat>   # materialise on main
phlo-transform ref create ci/pr-1 --from main
phlo-transform run --ref ci/pr-1                     # candidate adopts main's
                                                     # verified outputs
phlo-transform diff --from ci/pr-1 --to main
phlo-transform promote ci/pr-1 --to main --check
phlo-transform ref delete ci/pr-1                    # drops phlo-owned catalog

State and caching

  • State store: SQLite (default, .phlo/transform/state.db) or Postgres (--state postgres://...). Both migrate additively on open — old databases need no manual surgery.
  • skip — this environment's record vouches and the live identity check confirms the relation still holds it.
  • cached — a matching record elsewhere vouches and this environment's target provably holds the same physical output; adoption runs no model SQL.
  • build — nothing vouches; the model executes.
  • Records from adapters that cannot prove physical identity (DuckDB, non-Iceberg Trino) never authorise reuse — fail closed by design.

Lineage, contracts, diffs

  • lineage <model-or-column>, impact <target>; lineage --diff compares graphs across git refs.
  • Contracts declared in config; breaking changes gate promotion; declared renames (renamed_from) are not breaking.
  • diff compares candidate vs base data with keyed policies, tolerances, sampling and Iceberg $partitions metadata.

dbt migration

phlo-transform -r <dbt-project> translate --from dbt --check
phlo-transform -r <dbt-project> translate --from dbt --out generated/ --verify

ref()/source()/var(), materialisations, keys, schema tests, tags and config inheritance are lowered to directives + phlo.toml/transform.toml.

Machine API

phlo-transform daemon --port 7070 serves /v1/... reads and POST /v1/operations for mutations with idempotency keys and /v1/operations/{id}/cancel. /v1/status reports version, semantics version, counts and capabilities. --token gates everything except liveness behind Authorization: Bearer.

Known limitations

These are real and deliberately documented — none are silent:

  • Warm plans read one Iceberg snapshot per model. At 5,000 models a warm plan is ~28s against a single Nessie/Trino pair (--since stays sub-second). This is the cost of an honest drift check.
  • Postgres state has no TLS (NoTls) — use it on trusted networks.
  • Nessie Iceberg catalogs cannot store views — Nessie-targeted projects must materialise views as tables.
  • dbt dynamic Jinja is deliberately limited — static config/var/ ref/source translate; macros, snapshots and dynamic control flow are classified REVIEW/UNSUPPORTED rather than miscompiled.
  • No cross-adapter or cross-engine cache reuse — reuse is Phlo/Iceberg/Nessie only, by design.
  • plan --ref on a never-provisioned candidate previews an empty target; run --ref provisions first and may adopt.
  • The daemon reloads by full recompile (no targeted invalidation), and its responses are more terse than the CLI's.

Installation

cargo install --path crates/phlo-transform-cli --locked
phlo-transform --version   # phlo-transform 0.1.0

No external services are required for the DuckDB path — the adapter is embedded. Prebuilt binaries are attached to the GitHub release for x86_64-unknown-linux-gnu, aarch64-unknown-linux-gnu, aarch64-apple-darwin, x86_64-apple-darwin and x86_64-pc-windows-msvc. Each archive has a matching SHA-256 checksum file.