fix(docker): support non-root runtime execution

[ko3n1g]

Background / motivation

• ARC and sandbox callers may run the published image as a non-root identity.
• The image installed uv and managed Python below /root, and runtime paths were root-owned.
• Follow the non-root runtime contract established by Automodel #3484 and Megatron-LM #6421.
• Preserve root as the published image default.

What changed

• Install pinned uv and uvx release archives in /usr/local/bin after architecture-specific SHA-256 verification, with managed Python under /opt/uv/python.
• Add configurable RUNTIME_UID and RUNTIME_GID build arguments, defaulting to 65532:65532.
• Conditionally create the runtime group/user and /home/nemo-runtime.
• Keep the preserved Git checkout root-owned during package installation, then transfer the completed source tree and mutable runtime paths to the runtime identity.
• Separate the root-owned build cache from writable runtime uv/XDG caches.
• Validate the non-root contract during the image build, then restore USER root.
• Exclude local .venv content from Docker build contexts.
• Allowlist the pinned uv archive checksums for the change-set secrets scan.

Details

• Runtime callers opt in with --user 65532:65532 and explicitly set HOME=/home/nemo-runtime and XDG_CACHE_HOME=/opt/nemo-gym/cache/xdg.
• The symlinked project environment continues to read packages from the root-owned build cache at /opt/uv/cache.
• Runtime uv state uses /opt/nemo-gym/cache/uv; Gym cache, results, source, and prefetched environments are owned by UID/GID 65532.
• The build-stage smoke test verifies:
  • UID 65532 execution.
  • uv, uvx, and the project interpreter.
  • Home, XDG, uv, source, cache, results, and site-packages writes.
  • Removal of a root-created site-packages probe.
  • gym --help.
  • Python 3.13 virtualenv creation and execution with uv.
• The final Dockerfile instruction restores USER root.
• detect-secrets flagged both uv_sha256 literals as Hex High Entropy String. Each is preceded by a column-0 # pragma: allowlist nextline secret line; the inline same-line form is unusable here because BuildKit joins RUN continuation lines into a single shell command, so a trailing # would comment out the remainder of the instruction.

Tested

• docker buildx build --check --build-context nemo-gym=. --target release -f docker/Dockerfile .
  • Check completed with no warnings at source 52e5eac7d1f917b43622cbe6a2f04a0a2c091b13.
• docker buildx build --builder multiarch --progress=plain --build-context nemo-gym=. -f docker/Dockerfile --tag nemo-gym-rootless:pr2560 --load .
  • Release image build and embedded UID 65532 smoke test passed.
  • Loaded image: nemo-gym-rootless:pr2560.
• docker image inspect --format {{json .Config.User}} nemo-gym-rootless:pr2560
  • Observed configured user: root.
• docker run --rm --entrypoint id nemo-gym-rootless:pr2560 -u
  • Observed UID: 0.
• docker run --rm --user 65532:65532 --env HOME=/home/nemo-runtime --env XDG_CACHE_HOME=/opt/nemo-gym/cache/xdg nemo-gym-rootless:pr2560 --help
  • Exited successfully.
• Runtime UID/write checks against the loaded image:
  • Observed runtime-user=65532.
• uv run --locked --extra dev pytest tests/unit_tests/ -q
  • 2,461 passed, 41 skipped, 15 subtests passed.
• uv run pre-commit run --all-files
  • All hooks passed.
• CI-Events OCI-builder workload oci-90e81a1077c4365e79891ccf:
  • Built exact source 52e5eac7d1f917b43622cbe6a2f04a0a2c091b13 for amd64.
  • Used the declared nvcr.io/nvidia/cuda-dl-base:26.03-cuda13.2-devel-ubuntu24.04 base through the documented flattened-filesystem path; no alternate base image was selected.
  • Gitleaks, DevSkim, Bandit, and LLM review passed.
  • OCI build, embedded non-root smoke test, registry push, archive, and terminal status passed.
  • Used the contract-permitted native fallback classified as gvisor-dpkg-hardlink.
  • Published immutable image: 766267172432.dkr.ecr.us-east-1.amazonaws.com/nemo-autobot/agent-sandbox@sha256:6cb5462321dda99f1eaadb273efe3a30b62806d91ec059b0f5f26b39ca189177.
  • ECR verified tag gym-pr-2560-rootless-52e5eac7-amd64 at the same digest; pushed 2026-08-15T01:55:48Z.
• detect-secrets-hook --baseline .github/workflows/config/.secrets.baseline over the CI change-set (git diff --name-only --diff-filter=d --merge-base origin/main)
  • Exit code 0 at source c4d47d7c3df4b547ea4c7b0f422cf5b714208cfb; previously exit 123 on docker/Dockerfile:83 and :87.
• BuildKit continuation probe (case block with the same column-0 comment lines)
  • Resolved command retained both assignments; observed PROBE arch=x86_64 sha=04f8b82f…be829.
• docker buildx build --check --build-context nemo-gym=. --target release -f docker/Dockerfile .
  • Check completed with no warnings after the change.

[thomasdhc]

Changes requested.

Review notes

• Server venvs resolve to <server_dir>/.venv under /opt/nemo-gym and the server uv cache to /opt/nemo-gym/cache/uv, which matches the runtime UV_CACHE_DIR this image now sets.
• global_config.py pins server venvs to the running interpreter version, so uv venv --python never has to write a new managed Python into the root-owned /opt/uv/python at runtime.

[thomasdhc]

[P2] Findings: this recursive chown runs in a layer after the gym env prefetch RUN that created /opt/nemo-gym/cache/uv and the per-server <server_dir>/.venv trees, and OCI layer diffs carry no metadata-only delta, so any build passing --build-arg NEMO_GYM_PREFETCH_CONFIGS=... rewrites that multi-GB content in full into a second layer. Proposal: apply the runtime ownership inside the prefetch RUN so this final RUN only covers the source tree, matching the find ... -type d treatment already used for /opt/nemo_gym_venv.

[thomasdhc]

LGTM.

Review notes

• Runtime ownership of /opt/nemo-gym/cache and prefetched server .venv trees is now applied inside the RUN that creates them, so prefetched layers are no longer duplicated by the final ownership pass.
• The in-build UID 65532 stage proves the CLI, uv toolchain, project environment, and cache/results/site-packages writability, and the final instruction restores root as the published default.