Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions .github/workflows/reusable-homebrew-bottle-maintenance.yml
Original file line number Diff line number Diff line change
Expand Up @@ -82,6 +82,7 @@ jobs:
kandelo-repository: Automattic/kandelo
kandelo-ref: main
tap-repository: Automattic/kandelo-homebrew
tap-name: Automattic/kandelo-homebrew
tap-ref: main
formulae: ${{ inputs.formulae }}
arches: ${{ inputs.arches }}
Expand Down
117 changes: 96 additions & 21 deletions .github/workflows/reusable-homebrew-bottle-publish.yml

Large diffs are not rendered by default.

4 changes: 4 additions & 0 deletions docs/architecture.md
Original file line number Diff line number Diff line change
Expand Up @@ -625,6 +625,10 @@ const restored = MemoryFileSystem.fromImage(image, { maxByteLength: 1024 * 1024
```

The image must also have been built with a large enough filesystem maximum, for example `MemoryFileSystem.create(sab, 1024 * 1024 * 1024)`. `fromImage(..., { maxByteLength })` only controls the restored buffer's runtime growth ceiling; `statfs`/`df` and allocation remain capped by the image superblock maximum.
Call `MemoryFileSystem.readImageCapacity(image)` when build tooling needs the
serialized buffer length and superblock ceiling without restoring the image.
Call `fromImagePreservingCapacity(image)` to restore a growable buffer with the
same ceiling the image builder recorded.

A consumer that must stage files larger than the image's recorded allocation
ceiling first calls `rebaseToNewFileSystem(requiredMaxBytes)`. Shared image
Expand Down
142 changes: 113 additions & 29 deletions docs/homebrew-publishing.md
Original file line number Diff line number Diff line change
@@ -1,9 +1,9 @@
# Homebrew Publishing

Kandelo's Homebrew publishing path is a first-party bottle publication and
validation pipeline. The implementation lives in the main
`Automattic/kandelo` repository; the live tap repository is
`Automattic/kandelo-homebrew`.
Kandelo's Homebrew publishing path is a bottle publication and validation
pipeline shared by the first-party tap and conventional third-party taps. The
implementation lives in the main `Automattic/kandelo` repository; the
first-party live tap repository is `Automattic/kandelo-homebrew`.

This is not a general user-facing Homebrew install guide yet. Do not document
`brew tap` or guest `brew install` commands until guest Homebrew install has
Expand All @@ -26,14 +26,21 @@ Homebrew's `bottle do` block.
|---|---|
| `Automattic/kandelo` | Schemas, validators, reusable workflows, package build scripts, VFS planner/builder, Node/browser smoke tests, and this documentation. |
| `Automattic/kandelo-homebrew` | Tap state: `Formula/`, generated `Kandelo/` sidecars, bottle blocks, and provenance reports. |
| `<owner>/homebrew-<name>` | A third-party tap's Formulae, generated state, GHCR bottle packages, and caller-scoped publication authority. |

The checked-in `homebrew/kandelo-homebrew/` directory is a reviewable template
and test fixture for the tap shape. Live generated tap state belongs in
`Automattic/kandelo-homebrew`, not in the main repository template.

Use the full repository name in automation and documentation. The chosen tap
name intentionally differs from Homebrew's common `homebrew-<name>` repository
convention, so do not infer a short tap alias without verifying it.
Repository identity and Homebrew tap identity are separate inputs. A
conventional repository `<owner>/homebrew-<name>` has canonical Homebrew tap
name `<owner>/<name>`. Repository identity owns GitHub checkout, GHCR paths,
and the caller token; tap identity owns `brew` references, installed Formula
paths, receipts, OCI titles, and Kandelo sidecars. The first-party repository
is an explicit exception: both its repository identity and tap name are
`Automattic/kandelo-homebrew`. No conventional repository may derive that
protected first-party tap name, so repository and tap identities remain a
one-to-one publication boundary.

## Artifact Model

Expand Down Expand Up @@ -153,7 +160,7 @@ Ruby.

Publish same-tap runtime dependencies before their consumers. The bottle
builder resolves the selected Formula's recursive runtime closure in
topological order, filters it to `Automattic/kandelo-homebrew`, and installs
topological order, filters it to the selected canonical tap name, and installs
each dependency separately with `--force-bottle --as-dependency
--ignore-dependencies`. A missing Kandelo bottle therefore fails before the
consumer source build; Homebrew is not allowed to silently replace a prior
Expand Down Expand Up @@ -239,7 +246,7 @@ The reusable publisher is:
.github/workflows/reusable-homebrew-bottle-publish.yml
```

The tap may call it with:
The first-party tap may call it with:

```yaml
jobs:
Expand All @@ -251,10 +258,16 @@ jobs:
uses: Automattic/kandelo/.github/workflows/reusable-homebrew-bottle-publish.yml@<trusted-ref>
with:
tap-repository: Automattic/kandelo-homebrew
tap-name: Automattic/kandelo-homebrew
formulae: hello
arches: wasm32
```

A conventional third-party tap repository such as `Example/homebrew-tools`
uses the same caller shape with `tap-repository: Example/homebrew-tools` and
`tap-name: Example/tools`. The repository and tap name pair is validated before
any checkout or dry-run exit.

The caller grants the maximum permission ceiling. A write-capable publication
caller must grant `contents: write`, `packages: write`, and `actions: read`, but
the reusable workflow explicitly downgrades each job to its required subset.
Expand All @@ -268,24 +281,29 @@ schedule explicitly narrows itself to read scopes. PRs from untrusted forks must
not receive this caller ceiling; they can run schema and local build checks but
cannot invoke the trusted publisher.

Every call is fixed to a reviewed `repository_dispatch` workflow in
`Automattic/kandelo-homebrew@main`. Non-dry calls may come from
`publish-bottles.yml` or `maintain-bottles.yml`; dry calls must come from
`dry-run-bottles.yml`. The normal caller is displayed as
Every call is fixed to a reviewed `repository_dispatch` workflow on the target
tap repository's `main` branch, and the caller repository must exactly equal
the target tap repository. Non-dry calls may come from `publish-bottles.yml` or
`maintain-bottles.yml`; dry calls must come from `dry-run-bottles.yml`. The
first-party normal caller is displayed as
**Publish Kandelo bottles**; do not restore the narrower legacy **Publish hello
bottle** name. The three dispatch events are `publish-kandelo-bottles`,
`dry-run-kandelo-bottles`, and `maintain-kandelo-bottles`. Publish and dry-run
payloads must select at least one Formula and architecture; an absent or empty
selection is an error, not a successful no-op.
Write-capable publication is additionally fixed to `Automattic/kandelo@main`
and `Automattic/kandelo-homebrew@main`. The bottle root is never caller-selected:
All publication, including dry runs, is additionally fixed to
`Automattic/kandelo@main` and the caller tap's `main` branch. The bottle root is
never caller-selected:
the workflow rejects a non-empty `bottle-root-url` and derives
`https://ghcr.io/v2/<lowercase-owner>/<lowercase-repository>` from the tap
repository. Arbitrary Kandelo or tap refs are accepted only by the reviewed
dry-run caller. The maintenance workflow is callable but is not directly
branch-dispatchable; its operator-facing caller must live on the protected
default branch and grant write scopes explicitly. Third-party actions in the
privileged path are pinned by commit.
repository. The separate reusable maintenance workflow remains first-party
specific because its rollback and deletion paths own default-tap state. A
third-party `maintain-bottles.yml` on the protected default branch may call the
generic publisher for rebuilds, but generic rollback and deletion orchestration
are not provided by this change. Third-party actions in the privileged path are
pinned by commit. The reusable workflow uses the caller's scoped
`github.token`; it cannot publish another repository's tap state or GHCR
packages because caller and target repository identities must match.

After a read-only planning job resolves the immutable Kandelo commit, tap
commit, ABI namespace, derived bottle root, and formula matrix, each
Expand All @@ -299,12 +317,15 @@ only per `(tap, formula)`, so unrelated Formulae retain parallel throughput:
`/home/linuxbrew/.linuxbrew` prefix. This preserves the selected prefix and
Cellar so ordinary host build-dependency bottles remain usable. Within that
read-only build, all Formula-evaluating Homebrew commands run as a distinct
unprivileged user. Kandelo, tap, and patched Homebrew source are recursively
non-writable and non-replaceable by that identity; only a root-provisioned
shared temporary root, Homebrew cache/temp, prefix, and build home are
writable. Dependency lists and install logs used by the workflow identity
live in a separate mode-0700 control directory under the protected output
root; Formula processes cannot preplant or replace those paths. The wrapper
unprivileged user. The original Kandelo and tap checkouts remain hidden from
that identity. Each transient service receives root-created, read-only bind
aliases for those exact trees, and the Kandelo SDK environment points only
at the alias. The patched Homebrew source is recursively non-writable and
non-replaceable; only a root-provisioned shared temporary root, Homebrew
cache/temp, prefix, and build home are writable. Dependency lists and install
logs used by the workflow identity live in a separate mode-0700 control
directory under the protected output root; Formula processes cannot preplant
or replace those paths. The wrapper
uses an explicit host `sudo` boundary, a fixed
environment allowlist, and a transient systemd service with control-group
kill semantics and `NoNewPrivileges=yes` for every Brew invocation. A final
Expand Down Expand Up @@ -575,28 +596,91 @@ those claims require the trusted publish and bottle validation paths below.
The shared planner is `planHomebrewVfs()` in
`host/src/homebrew-vfs-planner.ts`. It consumes `Kandelo/metadata.json` plus a
caller-provided link-manifest loader and rejects bad ABI, unsupported arch,
cache-key drift, missing packages, dependency cycles, unsafe paths, and
link-manifest bottle drift before any bottle bytes are extracted.
tap-identity drift, duplicate roots or metadata, cache-key drift, missing
packages, dependency cycles, unsafe paths, and link-manifest bottle drift
before any bottle bytes are extracted. It resolves the requested roots and
their single-tap dependency closure in deterministic dependency-first order.

The Node-side builder is `buildHomebrewVfs()` in
`host/src/homebrew-vfs-builder.ts`. It verifies bottle byte count and sha256,
extracts supported tar entries, stages kegs under the declared prefix,
validates receipts, applies link manifests, writes
`/etc/kandelo/homebrew-vfs.json`, and emits a build report.

The CLI starts with an empty VFS by default. Pass `--base-image` to overlay the
same verified bottle plan onto an explicit platform-only `.vfs` or `.vfs.zst`
base image. The base must declare the same kernel ABI as the bottle metadata
and must not already carry Homebrew composition metadata or
`/etc/kandelo/homebrew-vfs.json`; merging independently composed Homebrew
prefixes would lose package provenance, so it fails closed. Existing files
remain unchanged except for requested bottle/link paths and the builder-owned
Homebrew manifest (plus the optional profile fragment); path collisions fail
through the normal staging checks.

The output image metadata binds the exact base input with a bounded object:
SHA-256, byte count, and declared kernel ABI. It also records the selected tap
repository and canonical tap name. The JSON report carries the same binding
plus the base's full source metadata for auditing. Base signatures,
attestations, and other metadata are not copied onto the mutated output, and
large source metadata is not nested into each new image.

When `--max-bytes` is omitted, the builder restores the base with its recorded
filesystem maximum and does not rebuild existing inodes. Supplying a different
`--max-bytes` explicitly rebases the filesystem to that exact maximum, so
allocation and `statfs` agree with the requested capacity. Explicit maxima
must be multiples of the 4096-byte SharedFS block size.

For reproducible image composition, use the builder's static Brewfile subset.
For example:

```ruby
tap "automattic/kandelo-homebrew"
brew "sqlite"
brew "automattic/kandelo-homebrew/xz"
```

The subset accepts blank lines, comments, exactly one literal canonical
lowercase `tap "owner/tap"`, and between 1 and 128 literal `brew` entries.
Entries may use a bare formula name or a fully qualified name from that exact
tap. Bare and qualified forms normalize to the same root, so duplicates fail.
The selected tap must exactly match `metadata.json`, and the complete resolved
closure is limited to 128 packages.

The parser uses Ripper to inspect the syntax tree and never evaluates the file.
Options, interpolation, conditionals, variables, nested Ruby, `cask`, `mas`,
`service`, and every other Homebrew Bundle entry are rejected. Full Bundle DSL
belongs to real Homebrew running inside a Kandelo guest; it is not a safe or
deterministic host-side image specification. This builder intentionally accepts
only one tap, so a root or required dependency from another tap must fail until
multi-tap composition has an explicit metadata and provenance contract.

This path does not read `Brewfile.lock.json`; Homebrew Bundle does not define a
lock-file contract. Reproducibility instead comes from the exact Brewfile
SHA-256 and byte count, ordered normalized roots, tap commit, base-image digest,
and verified bottle digests recorded in the report and image manifest. The
root digest is SHA-256 over the UTF-8 JSON array of normalized roots in declared
order. The bounded top-level VFS metadata records only root count and digest
rather than embedding arbitrary source text.

Build a precomposed image with:

```bash
scripts/dev-shell.sh npx tsx images/vfs/scripts/build-homebrew-vfs-image.ts \
--metadata /path/to/kandelo-homebrew/Kandelo/metadata.json \
--tap-root /path/to/kandelo-homebrew \
--package hello \
--brewfile /path/to/Brewfile \
--arch wasm32 \
--runtime node \
--base-image target/platform-base.vfs.zst \
--out target/homebrew-hello.vfs.zst \
--report target/homebrew-hello.vfs-report.json
```

Repeatable `--package <name>` remains available for lower-level tooling and
focused tests. It preserves the provided root order and uses the same planner,
limits, and provenance report. `--package` and `--brewfile` are mutually
exclusive.

The bottle fetcher follows GHCR `WWW-Authenticate` bearer challenges. Public
bottle materializers do not need a GitHub token merely to read public GHCR
blobs.
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -15,6 +15,7 @@ jobs:
kandelo-repository: ${{ github.event.client_payload.kandelo_repository || 'Automattic/kandelo' }}
kandelo-ref: ${{ github.event.client_payload.kandelo_ref || 'main' }}
tap-repository: ${{ github.event.client_payload.tap_repository || 'Automattic/kandelo-homebrew' }}
tap-name: ${{ github.event.client_payload.tap_name || 'Automattic/kandelo-homebrew' }}
tap-ref: ${{ github.event.client_payload.tap_ref || 'main' }}
formulae: ${{ github.event.client_payload.formulae }}
arches: ${{ github.event.client_payload.arches || 'wasm32' }}
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -15,6 +15,7 @@ jobs:
kandelo-repository: Automattic/kandelo
kandelo-ref: main
tap-repository: Automattic/kandelo-homebrew
tap-name: Automattic/kandelo-homebrew
tap-ref: main
formulae: ${{ github.event.client_payload.formulae }}
arches: ${{ github.event.client_payload.arches || 'wasm32' }}
Expand Down
45 changes: 39 additions & 6 deletions homebrew/kandelo-homebrew/Kandelo/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -103,37 +103,70 @@ Host VFS tooling plans a Homebrew-prefix image with
shared by Node and browser callers. It consumes parsed `Kandelo/metadata.json`
and a caller-provided link-manifest loader, resolves requested packages plus
their dependency closure in dependency-first order, and rejects bad ABI,
unsupported arch, cache-key drift, missing packages, dependency cycles, unsafe
paths, and link-manifest bottle URL/sha/byte/cache-key drift before any bottle
bytes are extracted.
unsupported arch, tap-identity drift, duplicate roots or metadata, cache-key
drift, missing packages, dependency cycles, unsafe paths, and link-manifest
bottle URL/sha/byte/cache-key drift before any bottle bytes are extracted.

For `failed`, `pending`, or `building` bottle entries, the planner uses the
complete last-green fallback fields when available. Without a complete fallback,
the package is not plannable for a VFS image.

## VFS Image Building

Build a precomposed Homebrew-prefix image from generated sidecars and verified
bottle bytes with:
Write a static Brewfile that names one tap and the formula roots for the image:

```ruby
tap "automattic/kandelo-homebrew"
brew "sqlite"
brew "automattic/kandelo-homebrew/xz"
```

Then build a precomposed Homebrew-prefix image from generated sidecars and
verified bottle bytes with:

```bash
scripts/dev-shell.sh npx tsx images/vfs/scripts/build-homebrew-vfs-image.ts \
--metadata /path/to/kandelo-homebrew/Kandelo/metadata.json \
--tap-root /path/to/kandelo-homebrew \
--package hello \
--brewfile /path/to/Brewfile \
--arch wasm32 \
--runtime node \
--base-image target/platform-base.vfs.zst \
--out target/homebrew-hello.vfs.zst \
--report target/homebrew-hello.vfs-report.json
```

This is a deliberately small, non-executing Brewfile subset: blank lines,
comments, exactly one literal lowercase `tap "owner/tap"`, and 1 to 128 literal
`brew` entries. A formula may be bare or fully qualified under that exact tap;
duplicates after normalization are rejected. Ripper validates the syntax tree,
and the builder never evaluates the file. Options, interpolation, conditionals,
variables, nested Ruby, other entry types, and multi-tap selection are rejected.
Use real Homebrew inside a running Kandelo guest when full Homebrew Bundle DSL
behavior is required.

Homebrew Bundle does not define a `Brewfile.lock.json` contract. The report and
image manifest instead bind the Brewfile SHA-256 and byte count, ordered roots,
tap commit, base image, and exact bottle digests. The root digest is SHA-256 over
the UTF-8 JSON array of normalized roots in declared order. Repeatable
`--package <name>` is still available to lower-level callers, but it cannot be
combined with `--brewfile`.

The builder consumes only `metadata.json`, link manifests, and bottle tarballs.
It does not evaluate Formula Ruby. It verifies the selected bottle byte count
and sha256, rejects unsafe or unsupported tar entries, stages files under the
declared keg, validates receipts, applies the link manifest under the declared
prefix, writes `/etc/kandelo/homebrew-vfs.json`, saves a `.vfs.zst`, and emits a
JSON report beside the image.

`--base-image` is optional and accepts only an ABI-matched platform image that
does not already contain a Homebrew composition. Output image metadata records
a bounded binding with the base SHA-256, byte count, and declared ABI; the JSON
report also retains the full source metadata for auditing. Omit `--max-bytes`
to retain the base filesystem maximum without rebuilding existing inodes, or
set it to rebase the filesystem to an exact, 4096-byte-aligned new maximum
before bottles are staged.

Link and receipt paths starting with `Cellar/` are interpreted relative to the
Homebrew prefix. Other link and receipt paths are interpreted relative to the
staged keg. Bottle payload entries under `bottle.payload_root` map to the keg;
Expand Down
7 changes: 7 additions & 0 deletions homebrew/kandelo-homebrew/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -48,6 +48,13 @@ The protected dispatch events are `publish-kandelo-bottles`,
`dry-run-kandelo-bottles`, and `maintain-kandelo-bottles`; publish and dry-run
requests must include nonempty Formula and architecture selections.

The publisher keeps GitHub repository identity separate from canonical
Homebrew tap identity. Conventional third-party repository
`<owner>/homebrew-<name>` must pass tap name `<owner>/<name>` and must call from
that same repository's reviewed `main` workflow. This first-party repository is
an explicit naming exception: both values are
`Automattic/kandelo-homebrew`.

The caller grants the maximum permission ceiling. Four fresh runner roles then
downgrade it: a read-only build/test job, a `packages: write` uploader without
tap write access, a read-only anonymous/runtime verifier, and a
Expand Down
Loading
Loading