Skip to content

GQT (ordinary) times out cold and can never re-warm its cache (repo caches at 33 GB against the 10 GB limit) #755

Description

@ragnorc

Symptom

Every GQ Logic Tests run on a branch that changes GQT input fails since #742: the GQT (ordinary) matrix entry is cancelled at timeout-minutes: 45 and the aggregating GQ Logic Tests job reports qualification=cancelled. Seen on main (#742 push, run 35267399410), #751 (run 35339183519, both attempts) and #606 (run 35349793990). dst and dst-clippy pass in 2–3 minutes.

Mechanism

ordinary runs two builds with RUSTFLAGS="" (the flagless shape the dispatch refusal test exists for), which shares no artifacts with the workspace's --cfg tokio_unstable build:

run cache step 1 cargo test -p omnigraph-gqt --lib --test runner_dispatch step 2 cargo test -p omnigraph-engine --test failpoint_names_guard
#738 on main (last green, 24 min) Cache restored successfully (788 MB, partial match) Finished in 1m15s Finished in 21m36s
#742 on main No cache found. Finished in 27m07s (152 tests pass in 0.4 s) killed at 45:00 with rustc still running
#751, both attempts No cache found. 27m09s killed at 45:00

A job cancelled by its timeout never reaches Swatinem/rust-cache's post step, so no cache is saved (cache-on-failure: true does not cover cancellation). Once the cache is gone, every later run is cold, exceeds 45 minutes, and again saves nothing: the matrix entry cannot recover on its own.

Why it went cold: the repository holds 16 active caches totalling 33 GB against GitHub's 10 GB limit, so entries are evicted by recency. The gqt-ordinary cache is gone; only gqt-dst (827 MB) and gqt-dst-clippy (432 MB) remain, while five build_release caches (2.3–2.7 GB each), storage_upgrade_compatibility (3.6 GB), crossversion (2.8 GB) and test (2.3 GB) crowd it out. Cargo.lock, the toolchain and the crate manifests did not change in #742.

Fix

Any one of the first three breaks the loop; the fourth stops it recurring.

  1. Give ordinary a budget a cold build fits (its two steps cost about 27 + 22 minutes cold), or split its two flagless builds into two matrix entries so each fits 45 minutes and saves its own cache.
  2. Cut the cold cost: the seam guard's RUSTFLAGS: "" is not required by that test's purpose (the comment ties the flagless shape to the dispatch refusal test only); dropping it lets step 2 build under the default flags and share the dst entry's artifacts if the cache key is shared.
  3. Warm once by hand: run the job with a temporary longer timeout so a cache is saved, then restore the limit.
  4. Cache hygiene: save-if on main only for the multi-GB release and upgrade caches, prune with gh cache delete, and keep total usage under 10 GB so evictions stop hitting the small test caches.

Until one of these lands, #751 and #606 cannot go green on this job regardless of their content.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions