Skip to content

RELEASING.md

Latest commit

 

History

History
425 lines (354 loc) · 22.3 KB

RELEASING.md

File metadata and controls

425 lines (354 loc) · 22.3 KB
Raw

Releasing Stella

Stella ships prebuilt binaries, a Homebrew formula, and a curl | sh installer. Everything is driven by pushing a version tag — the Release workflow does the rest.

The workflow is a hand-rolled build matrix (it does not require cargo-dist on the runner). The [workspace.metadata.dist] block in the root Cargo.toml is retained for a possible future migration to dist but is not the active pipeline today; the source of truth is .github/workflows/release.yml.

The default path: every merge to main is a release

auto-tag.yml runs after ci goes green on main and does everything — no manual steps:

  1. Tags a release commit: one commit on top of the CI-validated merge commit carrying the version stamp (scripts/sync-versions.sh), reachable only through the tag — so the tagged tree reports its own version and any from-tag source build is correct with no build-time rewriting (#786). +1 patch by default; start the PR title with release:minor / release:major for bigger bumps, or include [skip release] to land without releasing.
  2. Dispatches release.yml at the tag (binaries, GitHub Release, tap formula — the workflow verifies the tagged manifest matches the tag and builds --locked).
  3. Writes the version back to main so the Cargo manifests stay in sync with the newest tag: a bot/version-sync PR bumps [workspace.package].version in Cargo.toml (every crate inherits it), the workspace-member entries in Cargo.lock, and packaging/homebrew/stella.rb, then merges once the required checks pass. Nobody has to touch it. That PR is written by github-actions[bot], and this repository makes a first-time contributor's workflow runs wait for a person to approve them, so all thirteen of its runs open parked at action_required. The workflow approves them itself and then waits for the checks, which is what lets the PR merge under branch protection rather than past it. Its commit carries [skip release], so the sync itself never cuts a release. If a sync PR is ever left open (red check, race with another merge), the next release supersedes it automatically — no cleanup needed. Both the stamp and the merge are checked against scripts/check-lockfile-sync.sh: the script refuses to author a stamp whose Cargo.lock does not resolve, and the workflow refuses to merge a sync branch that would leave main's lock unresolvable even though the branch's own checks were green (#3336). A refusal is the "left open" case above and heals on the next release — but "the next release" is a commit landing on main, not a guarantee, so a refusal on the last merge before a quiet period opens (or updates) one issue labelled version-writeback-deferred naming the branch, the version, and the lockfile reason, and closes it automatically the run the write-back finally merges (#3842). The merge itself is made with the token the workflow run holds, and GitHub starts no workflow for a push made with that token — so the sync commit lands on main with no ci run and no main-canary run of its own. The last step asks for them: scripts/dispatch-main-verification.sh counts the ci runs for main's tip and starts both workflows when there are none. On a minor or major release the same PR also rolls CHANGELOG.md: whatever sits under ## [Unreleased] moves beneath a new version heading, and [Unreleased] is left empty for the next line. A patch release does not touch that file — see below.

Manual version bumps are therefore only needed for the hand-cut flows below.

What records a change

  • CHANGELOG.md — the durable curated record, one section per minor line, and CI writes it, not PRs. When a minor or major release is cut, auto-tag.yml drafts the section from the whole series diff — the previous X.Y.0 tag to this one (scripts/changelog-ai.sh, the same AI Gateway key and model as the release notes) — and scripts/changelog-roll.sh injects it at both roll sites (the tagged release commit and the version-sync PR), so the tag and main record identical text. The draft replaces whatever sits under [Unreleased]: contributors and coding agents should leave that section alone rather than hand-write entries in a PR, because a split between hand-written and drafted entries is what used to make this file inconsistent.
  • GitHub Release notes — generated at publish time by release.yml from the commit range, for every tag including patches. Release-note prose, not a curated record, and not something to edit by hand.

A patch release records nothing in CHANGELOG.md. This is the important half of the split, and it is enforced in changelog-roll.sh rather than trusted to a caller. Every merge to main cuts a patch, so the roll used to fire ~130 times per minor line; changelog-ai.sh degrades open by contract (no API key, no non-bot commits, or an unparseable response all print nothing and exit 0), but the roll ran regardless and stamped a bare ## [0.6.x] — <date> heading with nothing under it. The file reached 180 sections of which 77 were empty. It was structurally guaranteed to fill with noise no matter how good the drafter got, so the roll stopped firing on patches — where the per-tag detail already had a home in the GitHub Release notes above.

The roll is deliberately non-fatal, and can no longer produce an empty or a duplicate section:

  • A missing CHANGELOG.md, or one with no ## [Unreleased] heading, logs a warning and continues. A bookkeeping slip must never be the reason a release fails to ship.
  • If the draft fails on a minor release and [Unreleased] is empty, the roll writes a pointer to the releases page rather than a heading with an empty body — the failure mode above, which would otherwise recur once per line.
  • It is idempotent. If the version already has a section, the roll leaves the file alone. That matters because it runs at two call sites per release, and because a maintainer may have written the section by hand in the release PR — a minor release is a considered event, and "CI writes this file" exists to stop per-PR bullets accumulating in inconsistent voices, not to overwrite a section someone sat down and wrote. Whoever got there first wins.

make changelog-roll-test (hermetic, not part of make gate) pins both rules.

One-time setup

The tag-triggered workflow publishes the Homebrew formula to a tap repo. This has to exist and be writable before the first release:

  1. Create the tap repo. A public repo named exactly macanderson/homebrew-tap (Homebrew maps the tap macanderson/tap → repo homebrew-tap). It can start empty; the release job commits Formula/stella.rb into it.

  2. Create write access, either way (the release job tries the deploy key first, falling back to the token — see .github/workflows/release.yml's homebrew job):

    • SSH deploy key (what's actually configured today) — generate a dedicated keypair, add the public half as a write-enabled deploy key on macanderson/homebrew-tap (repo Settings → Deploy keys), and the private half as the HOMEBREW_TAP_DEPLOY_KEY secret below. Scoped to exactly that one repo, unlike a PAT.
    • PAT (fallback) — a GitHub token with contents: write on the tap repo — a fine-grained PAT scoped to macanderson/homebrew-tap, or a classic PAT with repo. The default GITHUB_TOKEN can't push to another repo, so a dedicated one is required either way.

  3. Add it as a secret on this repo (macanderson/stella): Settings → Secrets and variables → Actions → New repository secret → name HOMEBREW_TAP_DEPLOY_KEY (deploy key) or HOMEBREW_TAP_TOKEN (PAT).

The prebuilt tarballs, checksums, and the curl | sh installer are published to this repo's GitHub Releases and need no extra secrets — only the Homebrew tap push does. If neither secret is set, the release still succeeds and the homebrew job skips with a warning.

Optional — RELEASE_ADMIN_TOKEN. The bot/version-sync PR merges on its own with no secret set, so this is not required. auto-tag.yml uses the token in two places, both as a second try after GITHUB_TOKEN:

  • Approving the PR's parked workflow runs. If GITHUB_TOKEN cannot approve them, the runs stay parked, no required check reports, and the PR waits for a person. The run log names which token did the approving.
  • Merging past branch protection. This no longer works on main, because enforce_admins is on and GitHub refuses an admin merge on the same grounds it refuses any other (#5589). It is kept for a repository that still grants a bypass.

To set it, add a classic PAT with repo scope, owned by an account with admin rights on this repository, as the RELEASE_ADMIN_TOKEN secret.

Cut a release

  1. Re-cut the demo video if the command deck moved — make record-demo-video, then watch it and commit docs/demo/. The film is a picture of the deck (doc:deck-film), so it goes stale exactly when the deck changes: a tab added, a panel moved, a status field renamed. Shipping a demo of a UI the download no longer has is a claim this project does not make. It needs no API key and takes about fifteen minutes. Watch it: the shot list frames rows the deck's own layout decides, so a layout change can move content out of shot without failing any check.

  2. Bump the version if needed — version in [workspace.package] of the root Cargo.toml (all crates inherit it) — and commit.

  3. Tag and push. The tag must be v<major>.<minor>.<patch> matching the workspace version:

    git tag v0.1.0
git push origin v0.1.0

That push starts the Release workflow, which:

  • builds stella for macOS (aarch64, x86_64) and Linux (aarch64, x86_64),
  • packs each as a stella-<version>-<target>.tar.gz with the licenses + README,
  • rebuilds x86_64-unknown-linux-gnu on a second runner and refuses to publish if the two binaries differ byte for byte (verify-reproducible),
  • creates a GitHub Release with those tarballs, a SHA256SUMS (tarballs) and a SHA256SUMS.bin (bare binaries),
  • renders Formula/stella.rb from .github/homebrew/stella.rb.tmpl (real version + per-target SHA-256 sums) and commits it to the Homebrew tap (skipped if HOMEBREW_TAP_TOKEN is not configured).

When a release fails

A failed release used to be silent. release.yml failed on 31 consecutive tags (v0.6.75 → v0.6.108, about two days) and nothing said so: ci on main was green, auto-tag succeeded — it tags, dispatches, and exits without looking at the outcome — the version-sync PR opened and auto-merged, CHANGELOG.md rolled, and every manifest was stamped. Only the last two jobs of release.yml skipped. Every surface a maintainer glances at said "released" (#1464).

Three checks now catch it, each on a different failure:

  • smoke (in release.yml) unpacks the artifact and runs it before anything is published, so a release that builds but does not work cannot reach the Homebrew tap (#1626).
  • release-reconcile.yml's reconcile job runs hourly and compares git tag -l 'v*' against gh release list, failing when a tag older than 90 minutes has no release. That window is deliberate: full-LTO plus the independent rebuild arm takes the better part of an hour, so "not published yet" is the normal state for a long time after every merge.
  • release-reconcile.yml's tap job asks the same question one hop later: does brew install macanderson/tap/stella actually serve the newest release? A release can publish perfectly and never reach the tap — the homebrew job warns and exits 0 when neither tap credential is set, so an expired deploy key or a revoked token freezes the formula while every other surface still reports success. Before this job, nothing anywhere could answer that question — which is its own hazard: a report that the tap is stale could be neither confirmed nor refuted without reading the tap by hand. It only ever fetches the newest 100 releases, so a tap stale enough to be behind every one of them is reported as "at least N releases behind" rather than a guessed exact count.

Run them yourself at any time:

make releases-published   # every tag has a release
make tap-current          # the tap serves the newest one

It compares state rather than watching the dispatched run, which catches strictly more — a dispatch that never started, a run killed by a runner outage, and a release deleted after the fact are all invisible to a watcher. It also deliberately does not read workflow conclusions: the release/homebrew jobs are gated on a tag ref, so "skipped" is the correct state for a workflow_dispatch on a branch and reading conclusions would report those as failures.

scripts/unpublished-tags-baseline.txt grandfathers the tags that shipped nothing and are not going to be republished (#1463), so the check is an alarm about new silence rather than a daily recital of a backlog. Adding to it is a decision — that tag will never have a release — and lands as a reviewable diff:

make releases-baseline-update    # then read the diff

Write the reason above the tag as a comment line. Regeneration keeps those notes, and the reason is what a reviewer needs to judge the entry.

The three shapes a failed release takes

make releases-published names the state of every reported tag, because the work each one needs is different. Four orphan tags once turned out to be three different failures:

State What happened What to do
draft The build ran and the assets are attached, but a 5xx killed the publish step before the release left draft. List the draft's assets. Publish it if the set is complete; delete it and re-run the tag if it is not.
absent No release object exists — a build job died, often in actions/upload-artifact. Re-run release.yml on the tag, but read the version guard below first.
gone The build finished, and the run itself was killed after the fact. Build artifacts expire after 7 days, so an old one has nothing left to publish. Grandfather the tag with a note. Rebuilding a version a hundred releases behind buys nothing.

Find a draft and see what it holds:

gh api repos/{owner}/{repo}/releases \
  --jq '.[] | select(.draft) | {id, tag_name, assets: [.assets[].name]}'

A complete set is four tarballs plus SHA256SUMS and SHA256SUMS.bin. install.sh and the Homebrew formula both read SHA256SUMS, so a draft missing it is not publishable — delete it (gh api -X DELETE repos/{owner}/{repo}/releases/<id>) rather than shipping a release whose checksum file is absent.

Re-running an old tag runs the workflow as it stood at that tag. That matters most for the homebrew job, which renders Formula/stella.rb for whatever version it is building. Today's job refuses to render a version lower than the one the tap already serves, so a re-run cannot downgrade brew upgrade. A tag old enough to predate that guard has no such protection: check what its own release.yml does before dispatching it.

Verify a published binary yourself

Release builds are reproducible: the same tag, built on the pinned toolchain, produces the same bytes on your machine as on ours. That is what makes the published binary checkable against the published source rather than merely asserted to match it (#910).

git clone --depth 1 --branch v<version> https://github.com/macanderson/stella
cd stella
rustup show                       # must report the rust-toolchain.toml pin
./scripts/repro-build.sh x86_64-unknown-linux-gnu

curl -fsSLO https://github.com/macanderson/stella/releases/download/v<version>/SHA256SUMS.bin
grep "stella-<version>-x86_64-unknown-linux-gnu" SHA256SUMS.bin

The last line of repro-build.sh and the matching line of SHA256SUMS.bin must be identical. Before you file a mismatch:

  • Use scripts/repro-build.sh, not cargo build --release. The remapping that removes your $CARGO_HOME and rustup sysroot from the binary lives in that script and nowhere else — a bare cargo build bakes your home directory into the bytes and will never match.
  • Compare against SHA256SUMS.bin, not SHA256SUMS. The latter hashes the tarballs; rebuilding gives you a binary, not our archive of it. (The tarballs are reproducible too — scripts/package-tarball.py zeroes the mtimes, owners and gzip timestamp — but only when packed the same way.)
  • The toolchain pin is not advisory. repro-build.sh refuses to build on anything but the version in rust-toolchain.toml, because two rustc releases inline different std and no amount of path remapping makes their output agree.

cargo install --git (install.sh's source fallback) is a different build path with no reproducibility promise — it does not go through this script.

Cut a release locally (no CI) — the degraded path

This path ships unattested artifacts. Prefer the tag-driven CI release above whenever Actions can run at all.

release.yml attests every tarball and the SHA256SUMS file with actions/attest-build-provenance — a Sigstore bundle bound to that workflow at that commit, which nobody holding the release can reissue. A local release cannot produce one: attestation needs the OIDC token only a GitHub Actions job can mint.

The downgrade is invisible to users. install.sh deliberately separates "no verifier available" from "verifier said no", and an unattested release lands in the first bucket — gh attestation verify reports no attestation, the installer logs one info line and installs anyway. Anyone running STELLA_REQUIRE_PROVENANCE=1 will instead be refused outright.

The bytes differ too: the Linux tarballs here are cargo-zigbuild cross compiles against glibc 2.17, where release.yml builds them natively. Same tag, different artifacts, depending on who cut it.

So the script refuses by default and requires the decision to be explicit.

When GitHub Actions is unavailable (e.g. an org billing hold) and a release still has to ship, scripts/release.sh does the entire pipeline from your Mac — build all four targets, publish the GitHub Release, and push the Homebrew formula — with the version auto-incremented:

git checkout main && git pull        # release exactly what's on origin/main
ALLOW_UNATTESTED=1 scripts/release.sh patch   # 0.1.15 -> 0.1.16 (also: minor, major)

Without ALLOW_UNATTESTED=1 the script stops before touching anything. When you do use it, say so in the release notes, so users understand why STELLA_REQUIRE_PROVENANCE=1 rejects that version.

It refuses to run unless your checkout is clean and matches origin/main, and it never leaves your tree modified. macOS targets build natively; the Linux targets cross-compile via cargo-zigbuild (Zig as the C/C++ cross-linker — no Docker, and it compiles the workspace's bundled C deps (SQLite, tree-sitter grammars) cleanly, unlike the old cross image). zig and cargo-zigbuild are auto-installed if missing. All release assets are uploaded in one call, which matters because this repo has immutable releases enabled — a published release's assets can't be added or changed afterward, so an incomplete release means cutting a new version.

The engine container

Every release tag also publishes packaging/docker/Dockerfile.serve as ghcr.io/macanderson/stella-serve:<version> and :latest, for linux/amd64 and linux/arm64 (.github/workflows/docker-serve-publish.yml). Each architecture is built on its own runner, smoked with scripts/smoke-serve-image.sh before its digest is pushed, and the two digests are tagged together at the end, so a tag never resolves on one architecture and not the other. A host that embeds the engine pins the version tag; latest is for a laptop. The image takes STELLA_SERVE_TOKEN or STELLA_SERVE_TOKEN_FILE at run time and nothing else.

After the release — how users install

Homebrew (prebuilt binary, no Rust toolchain):

brew install macanderson/tap/stella
# equivalently: brew tap macanderson/tap && brew install stella

Shell installer (macOS/Linux, no Homebrew):

curl -fsSL https://raw.githubusercontent.com/macanderson/stella/main/install.sh | sh

The installer detects the platform, downloads the matching tarball from the GitHub Release, verifies it against SHA256SUMS, and falls back to cargo install where no prebuilt binary matches. Upgrades are brew upgrade stella or re-running the installer.

brew reporting an old version is usually a stale local clone, not a stale tap. A tap is a git clone on the machine, and brew info reads whatever commit that clone is sitting on — it does not fetch. Run brew update first, then compare against make tap-current, which reads the tap over the API and answers whether the published formula is behind. A clone left unfetched for a few days can be dozens of releases behind a tap that is perfectly current.

Add the tap as macanderson/tap and nothing else. GitHub still redirects the repo's former name, so brew tap macanderson/stella <url> succeeds and leaves the machine with two taps for one repo. They fetch independently, so one goes stale while the other is current, and brew info then answers differently depending on which name was asked. brew untap macanderson/stella retires the duplicate; stella self-driving upgrade does it too.

Formula templates

Two formulas live in the repo, for two different purposes:

  • .github/homebrew/stella.rb.tmpl — the template the release workflow renders and pushes to the tap as Formula/stella.rb. Installs the prebuilt binary per platform. Edit this to change what lands in the tap.
  • packaging/homebrew/stella.rb — a build-from-source formula for local use (brew install --build-from-source ./packaging/homebrew/stella.rb); it compiles with cargo and needs no per-release sha maintenance.