fix(ci): let the fleet-agent release builds run to completion - #712
Conversation
 |
![greptile-apps[bot]](https://avatars.githubusercontent.com/in/867647?s=60&v=4){kind=small}
![pullfrog[bot]](https://avatars.githubusercontent.com/in/1768019?s=60&v=4){kind=small}
There was a problem hiding this comment.
✅ No new issues found.
Reviewed changes — one file, .github/workflows/release-fleet-agent.yml. I verified the PR's premise against the job data of the actual failed run (34577407167) rather than taking the body's word for it; every timing and cache claim checks out.
- Job timeout caps raised (30→60m macOS, 20→30m Linux) — repositioned as hung-runner backstops, with a comment recording why no cache sits above them. Healthy cold builds measured 9m02s (macOS arm64), ~5m21s–5m51s (Linux); the new caps give 3–4× headroom over the slowest observed leg.
- Step-level build timeouts dropped — the Intel leg timed out exactly at its 15m step cap (build step 08:05:59 → 08:21:11) while still compiling, which alone broke the v0.1.5 release:
Upload to CDNandPublish Release Assetsbothneedsthe whole matrix and were skipped. The job cap is the only bound these builds need. - Both
Cache cargosteps deleted — correct call. The workflow only runs on tag refs; GitHub restores caches from the run's own ref or the default branch only; and nothing onmasterwrites the*-cargo-fleet-agent-*prefix (grepped all workflows), so each entry was a guaranteed miss plus a 342 MB write per release — against a cache budget the author's own follow-ups show is already over its 10 GB cap. No danglingsteps.cachereferences remain; the retainedmusl-crosscache and itscache-hitguard are untouched.
Verification: YAML parses clean and actionlint reports no diagnostics on the changed lines. (1.7.7 does flag the pre-existing macos-26/macos-26-intel runner labels as unknown — stale bundled label DB; both labels demonstrably ran in the real failed run.) The fleet-agent-v0.1.5 workflow_dispatch re-run after merge, noted in the PR body, is the right remaining step — the workflow builds needs.version.outputs.tag, so dispatching from master still builds the tag's tree.
openai-compatible/@cf/zai-org/glm-5.3-flash ｜ 𝕏
The macOS build step's own 15m cap killed the Intel job mid-dependency on v0.1.5, skipping the CDN upload and publishing a release with no assets. Job caps are the only bound these builds need. The cargo caches went with it. They keyed on a prefix nothing outside this workflow writes and saved under the release tag's own ref scope, so no release could ever restore one — every build was already cold, at the cost of a 342 MB write per release against a cache budget that is already at its 10 GB limit.
18382b2 to
3c04ce9
Compare
Problem
The
fleet-agent-v0.1.5release published with no assets. TheBuild macOS amd64job hit the build step's owntimeout-minutes: 15while still compiling dependencies, which failed the job and skipped bothUpload to CDNandPublish Release Assets:Run: https://github.com/arcboxlabs/arcbox/actions/runs/34577407167
latest.jsonstill reportsv0.1.4and everyv0.1.5/arcbox-fleet-agent-*URL 404s, so the gateway pin cannot move to 0.1.5.Why 15m was never enough
All four builds in that run were cold — every
Cache cargostep completed in 1–2s, restoring nothing:macos-26)macos-26-intel)The Intel runner needs roughly twice the Apple-silicon time for the same cold tree, so its cap was the tightest one against the slowest machine.
The cache could never have helped
The only fleet-agent cache entry that exists:
A run restores caches from its own ref plus the default branch. This workflow only runs on tags, and nothing on
masterwrites the*-cargo-fleet-agent-*prefix, so each release saved into a scope the next release cannot read. Guaranteed 0% hit rate, plus a 342 MB write per release against a cache budget already at its 10 GB ceiling.Warming it on
master(thewarm-release-cache/cache/restoresplit thatci.ymlandrelease.ymlalready use) would not pay off here either:upload-cdnandpublishbothneeds: [version, build-macos, build-linux], so the release gates on the slowest leg. In the v0.1.4 run arm64 finished at 08:59:01 and sat idle until Intel completed at 09:04:16 — publish started 3s after Intel. Speeding up arm64 saves no release wall-clock.Changes
timeout-minutesfrom both build steps. The job caps are the only bound these builds need.Cache cargosteps, with comments recording why there is no cache here.Install Protobuf compilerkeeps its 5m cap (guards a network hang on a 10x-billed runner), and themusl-crosstoolchain cache stays (tag-scoped too, but its miss path is a 6s download).Verification
actionlint .github/workflows/release-fleet-agent.ymlpasses. Not exercised against a real release —fleet-agent-v0.1.5needs aworkflow_dispatchre-run after this merges. The workflow buildsneeds.version.outputs.tagrather than the dispatch ref, so dispatching frommasterstill builds the tag's tree.Follow-ups (not in this PR)
master-scoped warm entries thatrelease.ymldepends on disappear too. Apull_request: closedcleanup workflow would fix it permanently.macOS-cargo-release-*encodes no arch.runner.osismacOSon both arm64 and Intel, so an Intel job saving under that key would silently poison the arm64 release restore. Nothing does today; the key shouldn't leave it possible.macos-26would take Intel off the critical path entirely and collapse the matrix to one job. Nothing in the workflow executes the binary, so it only codesigns and uploads — worth a look if release wall-clock matters.