[CI]: Reclaim agent disk in premerge pipeline to fix ENOSPC - #264
wuhang2014 wants to merge 2 commits into
Conversation
Shared cpu_queue_premerge agents ran out of disk (e.g. build 799): the Docker Build step stranded a multi-GB vllm-router:<commit> image plus build cache on every build, cargo jobs left multi-GB target/ dirs in the mounted checkout, and the P/D GPU job left a ~20GB .venv (vllm + torch cu130) and target/ in its checkout. - Docker Build: prune stale vllm-router:* tags, dangling layers and builder cache before building; smoke-test the image and rmi it after (local tags are unused downstream; release images are pushed by release-pipeline.yml) - cargo jobs (clippy, tests, wheel, python): rm -rf target afterwards - Build Rust: keep only the release binary for the artifact upload - P/D GPU job: pre-clean and trap-rm .venv and target/ so failures still free ~25GB Signed-off-by: WU Hang <whlbx@hotmail.com>
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 44c77cf37b
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
| cargo clippy --all-targets --all-features -- -D warnings | ||
| # Free agent disk: target/ holds multi-GB of build artifacts | ||
| # that are never reused across jobs (separate checkouts). | ||
| rm -rf target || true |
There was a problem hiding this comment.
These Docker plugin commands explicitly invoke bash -c without -e, so if Clippy fails, execution continues and this successful rm -rf ... || true becomes the shell's exit status, making the check pass. The same failure masking occurs in the newly appended cleanup commands for the release build, wheel build, unit tests, integration tests, and Python tests; use an EXIT trap that preserves the original status (or otherwise capture and return it) so cleanup cannot turn failed CI checks green.
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
🟡 Changes recommended
The new Docker cleanup command uses {{...}} in docker images --format, which can be consumed by Buildkite templating and break the cleanup, undermining the ENOSPC fix.
Once you've addressed the issues Copilot identified, you can request another Copilot review.
Pull request overview
Updates the Buildkite premerge pipeline to proactively reclaim disk space on long-lived shared agents (CPU and GPU queues), addressing ENOSPC failures caused by accumulated Docker images/build cache and Rust/Python build artifacts.
Changes:
- Add
target/cleanup after cargo-based CI steps and prune intermediate release artifacts while keeping the uploaded binary. - Add GPU e2e workspace hygiene (
.venv/targetpre-clean +trap ... EXIT) to reclaim space even on failures. - Add Docker daemon hygiene (prune stale vllm-router images/cache pre-build, smoke-test the built image, then remove it).
File summaries
| File | Description |
|---|---|
| .buildkite/pipeline.yml | Adds per-step cleanup of Cargo artifacts, GPU venv cleanup with traps, and Docker image/cache pruning + post-build image removal to prevent disk exhaustion. |
Review details
Suppressed comments (1)
.buildkite/pipeline.yml:486
- If the smoke test (
docker run ... --version) fails, the script may exit before reaching thedocker rmi ...line, leaving the per-commit image on disk (the exact leak this PR is addressing). Add anEXITtrap before the smoke test so cleanup runs even on failure.
# Smoke test the image, then drop it: keeping one multi-GB image per
# commit fills the agent disk.
docker run --rm --entrypoint /bin/sh vllm-router:latest -c 'vllm-router --version'
docker rmi vllm-router:${BUILDKITE_COMMIT} vllm-router:latest || true
- Files reviewed: 1/1 changed files
- Comments generated: 1
- Review effort level: Lite
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
| docker images --format '{{.Repository}}:{{.Tag}} {{.ID}}' \ | ||
| | awk '$1 ~ /^vllm-router:/ && $1 != "vllm-router:latest" {print $2}' \ | ||
| | xargs -r docker rmi -f || true |
The pipeline uploader's env interpolation (buildkite-agent 3.73.1) rejects dollar-identifiers that start with a digit, so the awk positional args in the pre-build docker cleanup aborted pipeline upload entirely (build 801: 'Expected identifier to start with a letter, got 1'). It scans the whole command string, comments included. Replace the awk/--format pipeline with 'docker images -q vllm-router | xargs -r docker rmi -f': all vllm-router images can go since latest is re-tagged in the same step anyway. Verified with buildkite-agent 3.73.1 locally. Signed-off-by: WU Hang <whlbx@hotmail.com>
hsliuustc0106
left a comment
hsliuustc0106
left a comment
There was a problem hiding this comment.
[P1] Cleanup turns failed CI checks green (.buildkite/pipeline.yml:58; also lines 98, 124, 158, 179, and 205).
The six affected Docker plugin steps explicitly use bash -c without -e. Appending rm -rf ... || true replaces the build/test exit status with success, so failed Clippy checks, Rust builds, wheel builds, Rust unit/integration tests, and Python tests can pass CI. Please use an EXIT trap that preserves the original status, or explicitly capture and return the build/test status after cleanup.
I verified this against base 0519da5 and PR head 0a19209 using the actual command blocks with stubbed build/test commands returning 42: all six base scripts returned 42, while all six PR scripts returned 0. This confirms the existing inline finding remains unresolved at the current head.
Validation: YAML parsing and bash syntax checks passed for all nine plugin scripts. Full Docker/GPU CI was not run.
Purpose
Premerge builds on the shared agents have started failing with
No space left on device(e.g. build 799). The pipeline leaks disk on every build:Build Docker Imageruns a multi-stagedocker buildwhoseRUN cargo build --releaselayer is multi-GB, tags itvllm-router:<commit>, and never pushes or deletes it. Nothing consumes the local tags downstream (release images are built and pushed byrelease-pipeline.yml), so each build permanently strands multi-GB of tagged images + build cache oncpu_queue_premergeagents. After ~800 builds the disks fill up.target/dirs in the host-mounted checkout (.:/workdir). Intermediates are never reused across jobs (separate checkouts)..venv(vllm + torch cu130) plustarget/in the/workdircheckout ongpu_4_queue, even when the job fails.For reference, vllm-project/vllm CI avoids this class of problem by building images with buildx/bake into a registry with ECR layer cache, scheduling registry cleanup (
cleanup-nightly-builds.sh), and running on ephemeral agents — none of which the router pipeline can assume from within the repo, so this PR makes the pipeline self-cleaning instead.Changes (all in
.buildkite/pipeline.yml)vllm-router:*tags (exceptlatest), dangling layers, and builder cache (--keep-storage 10GB) before building — this also reclaims disk on agents that are already full; after building, smoke-test the image withvllm-router --versioninsidedocker runanddocker rmiit. Local tags are unused downstream.rm -rf targetafter the job.target/release/vllm-routerfor the artifact upload; deletedeps/build/.fingerprintintermediates..venv/targetat start andtrap ... EXITso ~25GB is freed even when the job fails.Test Plan
python3YAML parse of.buildkite/pipeline.yml— OKbash -nsyntax check of every modified command block — OKcheck-yaml) — passTest Result
Pipeline-level change only; behavior verified by YAML/bash syntax checks and the pre-commit hook. Effect will be observable on the next premerge runs: the Docker Build step now logs
df -h /before/after pruning.