Symptom
Every GQ Logic Tests run on a branch that changes GQT input fails since #742: the GQT (ordinary) matrix entry is cancelled at timeout-minutes: 45 and the aggregating GQ Logic Tests job reports qualification=cancelled. Seen on main (#742 push, run 35267399410), #751 (run 35339183519, both attempts) and #606 (run 35349793990). dst and dst-clippy pass in 2–3 minutes.
Mechanism
ordinary runs two builds with RUSTFLAGS="" (the flagless shape the dispatch refusal test exists for), which shares no artifacts with the workspace's --cfg tokio_unstable build:
| run |
cache |
step 1 cargo test -p omnigraph-gqt --lib --test runner_dispatch |
step 2 cargo test -p omnigraph-engine --test failpoint_names_guard |
| #738 on main (last green, 24 min) |
Cache restored successfully (788 MB, partial match) |
Finished in 1m15s |
Finished in 21m36s |
| #742 on main |
No cache found. |
Finished in 27m07s (152 tests pass in 0.4 s) |
killed at 45:00 with rustc still running |
| #751, both attempts |
No cache found. |
27m09s |
killed at 45:00 |
A job cancelled by its timeout never reaches Swatinem/rust-cache's post step, so no cache is saved (cache-on-failure: true does not cover cancellation). Once the cache is gone, every later run is cold, exceeds 45 minutes, and again saves nothing: the matrix entry cannot recover on its own.
Why it went cold: the repository holds 16 active caches totalling 33 GB against GitHub's 10 GB limit, so entries are evicted by recency. The gqt-ordinary cache is gone; only gqt-dst (827 MB) and gqt-dst-clippy (432 MB) remain, while five build_release caches (2.3–2.7 GB each), storage_upgrade_compatibility (3.6 GB), crossversion (2.8 GB) and test (2.3 GB) crowd it out. Cargo.lock, the toolchain and the crate manifests did not change in #742.
Fix
Any one of the first three breaks the loop; the fourth stops it recurring.
- Give
ordinary a budget a cold build fits (its two steps cost about 27 + 22 minutes cold), or split its two flagless builds into two matrix entries so each fits 45 minutes and saves its own cache.
- Cut the cold cost: the seam guard's
RUSTFLAGS: "" is not required by that test's purpose (the comment ties the flagless shape to the dispatch refusal test only); dropping it lets step 2 build under the default flags and share the dst entry's artifacts if the cache key is shared.
- Warm once by hand: run the job with a temporary longer timeout so a cache is saved, then restore the limit.
- Cache hygiene:
save-if on main only for the multi-GB release and upgrade caches, prune with gh cache delete, and keep total usage under 10 GB so evictions stop hitting the small test caches.
Until one of these lands, #751 and #606 cannot go green on this job regardless of their content.
Symptom
Every
GQ Logic Testsrun on a branch that changes GQT input fails since #742: theGQT (ordinary)matrix entry is cancelled attimeout-minutes: 45and the aggregatingGQ Logic Testsjob reportsqualification=cancelled. Seen on main (#742 push, run 35267399410), #751 (run 35339183519, both attempts) and #606 (run 35349793990).dstanddst-clippypass in 2–3 minutes.Mechanism
ordinaryruns two builds withRUSTFLAGS=""(the flagless shape the dispatch refusal test exists for), which shares no artifacts with the workspace's--cfg tokio_unstablebuild:cargo test -p omnigraph-gqt --lib --test runner_dispatchcargo test -p omnigraph-engine --test failpoint_names_guardCache restored successfully(788 MB, partial match)No cache found.No cache found.A job cancelled by its timeout never reaches
Swatinem/rust-cache's post step, so no cache is saved (cache-on-failure: truedoes not cover cancellation). Once the cache is gone, every later run is cold, exceeds 45 minutes, and again saves nothing: the matrix entry cannot recover on its own.Why it went cold: the repository holds 16 active caches totalling 33 GB against GitHub's 10 GB limit, so entries are evicted by recency. The
gqt-ordinarycache is gone; onlygqt-dst(827 MB) andgqt-dst-clippy(432 MB) remain, while fivebuild_releasecaches (2.3–2.7 GB each),storage_upgrade_compatibility(3.6 GB),crossversion(2.8 GB) andtest(2.3 GB) crowd it out.Cargo.lock, the toolchain and the crate manifests did not change in #742.Fix
Any one of the first three breaks the loop; the fourth stops it recurring.
ordinarya budget a cold build fits (its two steps cost about 27 + 22 minutes cold), or split its two flagless builds into two matrix entries so each fits 45 minutes and saves its own cache.RUSTFLAGS: ""is not required by that test's purpose (the comment ties the flagless shape to the dispatch refusal test only); dropping it lets step 2 build under the default flags and share thedstentry's artifacts if the cache key is shared.save-ifon main only for the multi-GB release and upgrade caches, prune withgh cache delete, and keep total usage under 10 GB so evictions stop hitting the small test caches.Until one of these lands, #751 and #606 cannot go green on this job regardless of their content.