Skip to content

feat(evaluator): Set up environment for custom Gym e2e test script - #2020

Draft
JashG wants to merge 2 commits into
mainfrom
custom-gym-preparation/jgulabrai
Draft

feat(evaluator): Set up environment for custom Gym e2e test script#2020
JashG wants to merge 2 commits into
mainfrom
custom-gym-preparation/jgulabrai

Conversation

@JashG

@JashG JashG commented Sep 11, 2026

Copy link
Copy Markdown
Contributor

Summary

PR 1 of 3 to add a manual end-to-end Gym Evaluation test script that enables testing and validating a built-in custom Gym environment against a Kubernetes deployment.

This PR focuses on the environment setup portion of the script. It supports an out-of-the-box custom Gym environment, and wires up support for bringing your own environment.

Changes

Built-in ASCII Tree environment

  • Supports a built-in custom environment whose dependencies are packaged as a wheel. Includes all the relevant plumbing for generating and caching the required components of the custom environment.
  • This PR also ships its own custom dependency, rather than pulling an external dependency. This gets packaged in the wheel and imported by the resource server.

Bring-your-own custom environment

  • Supports validating and staging your own custom environment via CLI args. You are responsible for packaging external dependencies as wheels and supplying those as a wheels-v1 environment.
  • Automatically extracts relevant metadata from the environment used in later PRs to validate the job outcome.

Type of Change

  • Code change (feature, bug fix, or refactor)
  • Code change with documentation updates
  • Documentation only
  • Contributor tooling or automation
  • CI, build, or test infrastructure

Quality Gates

  • Tests added or updated for changed behavior
  • Existing tests cover changed behavior — justification:
  • Tests not applicable — justification: This adds a manually invoked validation workflow; targeted linting and type checks cover the preparation code.
  • Documentation updated for user-visible behavior
  • Documentation not applicable — justification: This adds a manually invoked validation workflow; not published anywhere.

Verification

  • Pull request title follows the repository's Conventional Commit format
  • Every commit includes an appropriate Signed-off-by: trailer
  • uv run pre-commit run -a passes, or any blocked checks are identified below
  • Targeted tests pass, or tests are marked not applicable above
  • No secrets, API keys, or credentials are included

Targeted validation:

  • Targeted ruff check and ruff format --check passed for the changed Python files.
  • Commit hooks passed Ruff, formatting, ty, DCO, and repository checks.
  • Full pre-commit validation was blocked only by the unavailable Studio lint-staged command in the isolated worktree.

Provide reusable package staging plus a bundled ASCII Tree fixture so custom wheel delivery can be reviewed independently from cluster orchestration.

Signed-off-by: Jash Gulabrai <jgulabrai@nvidia.com>
Signed-off-by: Jash Gulabrai <jgulabrai@nvidia.com>
@github-actions github-actions Bot added the feat label Sep 11, 2026
@JashG
JashG added this pull request to stack #2022 September 11, 2026 21:51
@github-actions

Copy link
Copy Markdown
Contributor
Suite Lines Covered Line Rate Branch Rate
Unit Tests 44046/56011 78.6% 62.2%
Integration Tests 27458/53280 51.5% 22.7%

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant