Skip to content

test: add GPU end-to-end CI coverage - #2662

Open
balasaajay wants to merge 3 commits into
NVIDIA-NeMo:mainfrom
balasaajay:gpu-e2e-tests
Open

test: add GPU end-to-end CI coverage#2662
balasaajay wants to merge 3 commits into
NVIDIA-NeMo:mainfrom
balasaajay:gpu-e2e-tests

Conversation

@balasaajay

Copy link
Copy Markdown

What does this PR do?

Adds an end-to-end GPU test using the infrastructure introduced in #2630 and the test foundation from #2575.

The test exercises:

Qwen on GPU → vLLM → Gym model server → simple agent → string_match verifier

  • Serves Qwen/Qwen2.5-0.5B-Instruct at a pinned Hugging Face revision.
  • Runs locally inside the CI container without inference-provider API keys.
  • Uses one GPU and temperature 0.
  • Verifies a real rollout receives reward 1.0, uses the expected model, and contains valid token usage and output.
  • Prefetches isolated Gym server environments during the container build.
  • Persists the Hugging Face cache between runs.
  • Uploads rollout, Gym, vLLM, model-readiness, and GPU diagnostic artifacts even when the test fails.
  • Adds bounded readiness checks, evaluation timeouts, and graceful process cleanup.

Checklist

  • I have read the contributing guidelines.
  • The change is focused; unrelated "drive-by" edits are tracked as separate issues/PRs.
  • Tests added or updated and pass locally, or N/A for docs-only / non-code changes (so CI unit/server checks pass when applicable).
  • Pre-commit checks pass locally (pre-commit run --all-files) (so CI lint/format/copyright pass).
  • All commits have DCO sign-off (git commit -s) (so the DCO check passes).

ananthsub and others added 2 commits August 21, 2026 13:47
Exercise the packaged CLI and isolated Gym components through deterministic CPU tool use.
Add a configurable GPU smoke lane for external vLLM runners.

Signed-off-by: Ananth Subramaniam <ansubramania@nvidia.com>
…n and artifact uploads

- Added optional inputs for model name, model revision, Hugging Face cache path, and results path in the test template action.
- Updated GPU E2E workflow to utilize the new model configuration and upload test artifacts.
- Removed obsolete CPU E2E workflow and related scripts.
- Enhanced GPU E2E test script to handle new parameters and validate outputs against expected results.

Signed-off-by: Ajay Balasa <abalasa@nvidia.com>
@balasaajay
balasaajay requested a review from a team as a code owner August 21, 2026 21:22
…riables

- Removed the optional Hugging Face cache path input from the test template action.
- Updated the GPU E2E workflow to eliminate references to the Hugging Face cache path.
- Modified the GPU E2E test script to use the new HF_HOME environment variable instead of HF_CACHE_DIR.
- Adjusted volume mounts and environment variables in the action and test script for improved clarity and functionality.

Signed-off-by: Ajay Balasa <abalasa@nvidia.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants