packaged-workflow gold path works end-to-end: compiler fixes, sandbox excision, canonical example, harness + CI coverage#192
Merged
Conversation
added 3 commits
July 10, 2026 04:42
…ge cleans both Root-causes the demo-stack flakiness. Three fixes: - angreal purge now tears down BOTH stacks (backing-services + the ui/demo stack). Previously it only downed .angreal/docker-compose.yaml, so the cloacina-demo stack survived purges for days and its DB pool exhausted. - Pin the backing-services stack to project name 'cloacina-dev' so it never collides with the default 'angreal' project used by other angreal repos. - Add 'max_connections=500' to the demo Postgres (the full fleet — server + N agents + compilers — exhausts the default 100; the likely cause of the 'Connection pool: Timeout waiting for a slot' we hit). The two stacks stay separate by design: the demo runs cloacina-server IN a container (kafka:9092, dex host.docker.internal), the test/services stack runs it on the HOST (localhost). Merging them would force dual kafka listeners + a split-horizon dex issuer — more complexity, not less.
…e with other projects' DBs The cloacina-dev stack claimed the postgres default 5432, colliding with other projects' databases on the same machine (opaque docker port-bind failures in the e2e harness). Publish 15432:5432 instead and point every host-side dialer at it: harness db_urls, Rust/python test fallbacks, the python conftest, Dockerfile.test, and nightly CI (service mapping, DATABASE_URL, macos setup-postgres). In-cluster URLs (compose service DNS, helm) and user-facing docs keep the standard 5432 — those are the user's own databases.
… — compiler --dev-workspace hatch + e2e coverage
A package authored the way cloacinactl package new emits it (version deps,
e.g. cloacina-workflow = "0.10") could not be built by any dev/e2e compiler:
the cloacina crates aren't published, so offline resolution failed — the
shipping dep form had zero e2e coverage while only the __WORKSPACE__
path-dep escape hatch was exercised.
Compiler: new --dev-workspace <root> flag (CLOACINA_COMPILER_DEV_WORKSPACE).
When set, each build injects [patch.crates-io] mapping <root>/crates/* to
their local paths, and the workspace root is bound read-only in the build
sandbox (bwrap + landlock) so the patched crates and their { workspace =
true } inheritance resolve. Strictly a dev-cycle escape hatch: unset (the
production default) nothing changes and packages resolve from crates.io.
Harness: angreal test e2e compiler --version-deps stages the happy fixture
in the version-dep manifest form and starts the compiler with
--dev-workspace — always compiling the LOCAL dev workspace, offline;
crates.io is never contacted. Both lanes verified green end-to-end
(build_status=success -> reconciler -> execution Completed), path-dep
lanes unaffected.
Example: simple-packaged re-authored to the canonical version-dep manifest;
demo compose compilers pass --dev-workspace /workspace.
dylanbstorey
changed the title
added 3 commits
July 10, 2026 20:14
… runner in the CI examples lane simple-packaged is now THE canonical example, run through the primary interface only (pack -> upload -> compile -> reconcile -> execute -> observe). The README is rewritten as that recipe against the demo stack — the stale content (broken cargo run --example commands, in-process registry/scheduler snippets, dead links) is gone, and there is no embedded-runner path; examples demonstrate features through the primary interface (I-0138 D-3). The harness orchestrates the whole flow: a bespoke `angreal demos features simple-packaged` command (the demos surface is the examples/tutorials path CI already runs) reuses the compiler-e2e lifecycle helpers — fresh dev-stack postgres + server + compiler with --dev-workspace — then drives cloacinactl pack/upload, polls the build to success, runs data_processing, and polls the execution to Completed. Wired into the rust-examples CI matrix (examples-docs.yml). Verified live: exit 0, execution Completed.
…ompile inside it offline The sandbox denies registry writes by design (SEC-06), but cargo's model wants network + a writable registry — so on a fresh demo stack EVERY Rust package build failed mid-download (Permission denied writing the read-only cargo home). Old stacks masked it with volumes that predated the sandbox. Split the build in two: - Phase 1: cargo fetch, UNSANDBOXED. Fetch resolves the graph and downloads crates; no build scripts or proc-macros execute, so trusted cargo code may use the network and write the registry. It also writes the Cargo.lock phase 2 builds against, so both phases see one dependency graph. - Phase 2: cargo build with --offline appended, fully sandboxed. Untrusted package code runs with the network cut and the registry read-only (landlock now grants explicit RO on CARGO_HOME for non-/usr layouts). Operator control is unchanged: passing --frozen/--offline in --cargo-flag (the curated --vendor-dir posture) skips the fetch phase entirely — the vendored registry stays authoritative. No new flags. Demo stack: compilers share a cargo-registry volume so fetches survive container recreations; no registry is baked into the image. Verified: angreal test e2e compiler --version-deps full pass (fetch + offline build; reconciler -> execution Completed; all failure/lifecycle lanes green).
…lation is the boundary Maintainer decision: the bwrap/landlock build sandbox (I-0105) is removed. Tenant-level isolation is Cloacina's security boundary — one compiler per tenant (--tenant-schema) keeps a tenant's source, logs, and artifacts separate — and per-build process sandboxing can return as an optional layer later if needed. In one day of real fresh-stack verification the sandbox surfaced three stacked defects (registry writes denied mid- download, landlock ABI::V1 denying every cross-hierarchy rename with EXDEV, the bwrap probe silently downgrading in containers) — it had never actually run a real compile before, and the cost/benefit no longer holds. Removed: sandbox.rs (probe, bwrap/landlock wrappers, env scrub), CompilerConfig.sandbox_level, the boot probe, CLOACINA_COMPILER_SANDBOX, compose seccomp=unconfined, bubblewrap in both Dockerfiles, Helm compiler.sandbox.* values + seccompProfile templating, the compiler-sandbox doc, and the bwrap escape e2e test. Kept (orthogonal): setrlimit resource ceilings, the wall-clock build timeout, --vendor-dir registry curation, the two-phase fetch->offline build (deterministic resolution; fetch errors separated from compile errors), and the --dev-workspace patch hatch. Audit events keep the sandbox_level field for schema stability and now truthfully report "none".
dylanbstorey
changed the title
added 3 commits
July 11, 2026 15:22
The rust-examples CI matrix was a hand-maintained list (5 of the 12 runnable feature demos) — exactly the drift that let examples fall out of coverage. Now a discover job runs `angreal demos matrix`, which emits the example set from the same discovery code that registers the `demos features <name>` commands, and the matrix is fromJSON of that output. A new example directory automatically joins CI; there is no list to maintain. 12 examples run today: conditional-retries, cron-scheduling, deferred-tasks, event-triggers, filtered-reactor, multi-tenant, packaged-graph, per-tenant-credentials, python-packaged-graph, python-workflow, registry-execution, simple-packaged. The harness now also owns the DB wiring explicitly: examples and python tutorials keep their user-facing localhost:5432 fallbacks (the user's own postgres), and the runners set DATABASE_URL to the dev stack's 15432 — previously the runners relied on the examples' fallbacks, which broke silently when the dev stack moved ports. Python tutorial 06 becomes env-aware (it dialed a hardcoded URL). Dirs still excluded from the matrix (complex-dag, packaged-workflows, packaged-triggers, validation-failures) aren't runnable-as-demos yet — that's the I-0138 migration backlog. Also files CLOACI-T-0888: DatabaseAdmin::build_connection_string hardcodes localhost:5432 in tenant credentials (latent product bug found during verification). Smoke-verified: angreal demos matrix emits the 12; demos features conditional-retries passes through the new env wiring.
…9..T-0893) Audit grounded against the real surfaces (cloacina-macros, the cloaca register_authoring contract, server routes, cloacinactl nouns): examples cover core authoring well but almost entirely through the embedded runner; via the primary interface only simple-packaged exists. Zero user-facing coverage: parameterized instances (I-0116), workflow secrets, boundary_schema, task->CG invocation, stream/batch/polling accumulators, constructors/providers (exist but outside harness+CI), and the whole operational verb surface. Five new tasks close the gaps; migration and gap-fill converge on one gold-path example per feature.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
This PR makes the packaged-workflow gold path (pack → upload → compile → reconcile → execute) actually work end-to-end, proves it on a fresh stack, and puts it under harness + CI coverage. It grew from a compose cleanup into the full fix because fresh-slate verification kept peeling the onion.
1. Compose-stack cleanup (original scope)
angreal purgetears down BOTH compose stacks; pinnedname: cloacina-dev; demo postgresmax_connections=500.2. Dev-stack postgres → host port 15432
kairos-postgres(another project) holds 5432 on the dev machine. The dev stack now publishes15432:5432; every harness db_url, Rust/python test fallback, and nightly CI follow. In-cluster URLs and user-facing docs keep 5432.3. Compiler:
--dev-workspacehatch (T-0887)Packages ship crates.io version deps (what
cloacinactl package newemits) — but the cloacina crates aren't published.--dev-workspace <root>injects[patch.crates-io]→ the local crates per build. Unset = production default, resolves from crates.io. The shipping dep form previously had zero e2e coverage.4. Compiler: two-phase builds
cargo fetchfirst (resolves + pins the lock, downloads), thencargo build --offline— deterministic single resolution, fetch errors separated from compile errors. Operators passing--frozen/--offline(curated vendor posture) keep single-phase.5. Build sandbox excised (maintainer decision)
Fresh-stack verification surfaced three stacked sandbox defects in one day (registry writes denied mid-download; landlock ABI::V1 denying every cross-hierarchy rename with EXDEV; the bwrap probe silently downgrading in containers) — it had never actually run a real compile. Tenant-level isolation is the security boundary (one compiler per tenant via
--tenant-schema); per-build process sandboxing can return as an optional layer later. Kept: rlimits, build timeout, vendor curation, two-phase. Audit events keepsandbox_level(now truthfullynone). Net −846 lines.6. Canonical example + harness + CI (T-0884 / I-0138)
simple-packagedre-authored to the version-dep manifest; README rewritten as the pure gold-path recipe (no embedded-runner path — I-0138 D-3).angreal demos features simple-packageddrives the whole lifecycle (fresh postgres + server + compiler + cloacinactl → execution Completed); added to therust-examplesCI matrix.angreal test e2e compiler --version-depscovers the shipping manifest form in the compiler harness.Verification
success— first time ever (previously all 13 Rust fixtures failed on any fresh stack).workflow run data_processing→ executionCompleted.angreal demos features simple-packaged→ pass (execution Completed).angreal test e2e compiler --version-deps+ default lane → full pass (happy/failed-build/content-hash/stale-heartbeat/reconciler/upgrade/rollback/concurrent).helm templaterenders; compose config validates; pre-commit (fmt, cargo check both backends, version lockstep) green on every commit.