Skip to content

test: add GPU end-to-end CI coverage - #2662

Merged
balasaajay merged 6 commits into
NVIDIA-NeMo:mainfrom
balasaajay:gpu-e2e-tests
Aug 24, 2026
Merged

test: add GPU end-to-end CI coverage#2662
balasaajay merged 6 commits into
NVIDIA-NeMo:mainfrom
balasaajay:gpu-e2e-tests

Conversation

@balasaajay

@balasaajay balasaajay commented Aug 21, 2026

Copy link
Copy Markdown
Contributor

What does this PR do?

Adds an end-to-end GPU test using the infrastructure introduced in #2630 and the test foundation from #2575.

The test exercises:

Qwen on GPU → vLLM → Gym model server → simple agent → string_match verifier

  • Serves Qwen/Qwen2.5-0.5B-Instruct at a pinned Hugging Face revision.
  • Runs locally inside the CI container without inference-provider API keys.
  • Uses one GPU and temperature 0.
  • Verifies a real rollout receives reward 1.0, uses the expected model, and contains valid token usage and output.
  • Prefetches isolated Gym server environments during the container build.
  • Persists the Hugging Face cache between runs.
  • Uploads rollout, Gym, vLLM, model-readiness, and GPU diagnostic artifacts even when the test fails.
  • Adds bounded readiness checks, evaluation timeouts, and graceful process cleanup.

Checklist

  • I have read the contributing guidelines.
  • The change is focused; unrelated "drive-by" edits are tracked as separate issues/PRs.
  • Tests added or updated and pass locally, or N/A for docs-only / non-code changes (so CI unit/server checks pass when applicable).
  • Pre-commit checks pass locally (pre-commit run --all-files) (so CI lint/format/copyright pass).
  • All commits have DCO sign-off (git commit -s) (so the DCO check passes).

ananthsub and others added 2 commits August 21, 2026 13:47
@ananthsub @balasaajay
test: add model-server end-to-end coverage
ab0a71c 
Exercise the packaged CLI and isolated Gym components through deterministic CPU tool use.
Add a configurable GPU smoke lane for external vLLM runners.

Signed-off-by: Ananth Subramaniam <ansubramania@nvidia.com>
@balasaajay
feat(gpu-e2e): enhance GPU end-to-end testing with model configuratio…
7f7cabe 
…n and artifact uploads

- Added optional inputs for model name, model revision, Hugging Face cache path, and results path in the test template action.
- Updated GPU E2E workflow to utilize the new model configuration and upload test artifacts.
- Removed obsolete CPU E2E workflow and related scripts.
- Enhanced GPU E2E test script to handle new parameters and validate outputs against expected results.

Signed-off-by: Ajay Balasa <abalasa@nvidia.com>
@balasaajay
balasaajay requested a review from a team as a code owner August 21, 2026 21:22
@balasaajay
refactor: streamline GPU E2E testing configuration and environment va…
be4cd8d 
…riables

- Removed the optional Hugging Face cache path input from the test template action.
- Updated the GPU E2E workflow to eliminate references to the Hugging Face cache path.
- Modified the GPU E2E test script to use the new HF_HOME environment variable instead of HF_CACHE_DIR.
- Adjusted volume mounts and environment variables in the action and test script for improved clarity and functionality.

Signed-off-by: Ajay Balasa <abalasa@nvidia.com>
@balasaajay
refactor: update GPU E2E test to use smoke configuration
0acc592 
- Removed references to the GPU E2E test script and model revision in the test case.
- Renamed the test function to reflect its focus on using the Qwen smoke configuration.
- Streamlined assertions to validate the workflow and configuration without the removed script dependencies.

Signed-off-by: Ajay Balasa <abalasa@nvidia.com>
@balasaajay
Merge branch 'main' into gpu-e2e-tests
207ab66
@kajalj22

kajalj22 commented Aug 24, 2026

Copy link
Copy Markdown
Contributor

/claude review

@claude

claude Bot commented Aug 24, 2026

Copy link
Copy Markdown
Contributor

SHIP — CI-only change; converts the placeholder nvidia-smi GPU E2E job into a real vLLM + Gym string_match rollout smoke test. Reviewed end to end.

Correctness checks that pass:

  • Smoke dataset (gpu_smoke.jsonl) schema matches the string_match verifier's StringMatchRunRequest contract exactly (expected_answer / extraction_mode=final_answer / case_sensitive at top level), confirmed against resources_servers/string_match/data/example.jsonl.
  • verify_gpu_rollout.py asserts on the actual rollout output shape — agent_ref={"name": "string_match_simple_agent"}, top-level reward / expected_answer / extracted_answer, response.usage.{input,output}_tokens — matching example_rollouts.jsonl. Not a vacuous check: it fails on wrong model, zero reward, empty text, and the unit tests exercise those rejection paths.
  • Every gym eval run flag the script uses exists in nemo_gym/cli/main.py (--no-serve, --agent, --input, --output, --limit, --concurrency, --temperature, --max-output-tokens, --model, --model-url, --model-api-key). The --no-serve -i combination is valid post-fix(eval): reject -i/--input in end-to-end eval run instead of silently ignoring it #2709 (only -i without --no-serve is rejected).
  • Default ports line up (vLLM 18000 ↔ config policy_base_url; head 11000 ↔ DEFAULT_HEAD_SERVER_PORT), cleanup trap has graceful TERM→KILL escalation with log tails on failure, and timeout --signal=INT --kill-after=30s bounds the eval. job timeout-minutes: 30 caps the whole thing.
  • test-template mount remap ($TEST_DATA_PATH:/home/TestData) is only consumed by the GPU matrix; no other test-template caller exists, so no CPU-path regression.

NOTE (author's call): gym env start pins ++head_server.port=$HEAD_PORT, but gym eval run --no-serve uses the default 11000 with no matching override. They agree only because HEAD_PORT defaults to 11000 — override HEAD_PORT and the eval silently targets the wrong head server. Consider passing ++head_server.port=$HEAD_PORT to the eval invocation too, or dropping the HEAD_PORT knob.

No reliability, async, or public-API concerns.

kajalj22
kajalj22 previously approved these changes Aug 24, 2026
View reviewed changes
@balasaajay
Merge branch 'main' into gpu-e2e-tests
550c5dc 
Signed-off-by: Ajay <abalasa@nvidia.com>
@balasaajay
balasaajay enabled auto-merge (squash) August 24, 2026 22:14
@kajalj22
kajalj22 self-requested a review August 24, 2026 22:16
@balasaajay
balasaajay merged commit 89918cb into NVIDIA-NeMo:main Aug 24, 2026
40 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@kajalj22 kajalj22 kajalj22 approved these changes

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@balasaajay @kajalj22 @ananthsub