fix(docker): make source readable by the runtime UID - #2632
Open
kajalj22 wants to merge 1 commit into
Open
Conversation
The release stage copies the full Git checkout without --chown, so the
source keeps whatever ownership and mode the build context supplied. The
ownership pass that follows deliberately covers directories only, to
avoid duplicating the source tree into another layer:
find /opt/nemo-gym ... -type d -exec chown ... {} +
That leaves the files themselves root-owned. It works whenever the build
context is world-readable, which is the case for the self-contained path
where the nemo-gym stage is populated by ADD from GitHub.
It does not hold for every builder. NeMo CI builds this image with
`--build-context nemo-gym=.` from a checkout made under `umask 0007`, so
the files arrive as mode 0660 owned by root. After `USER ${RUNTIME_UID}`
the interpreter can no longer read them:
File "/opt/nemo_gym_venv/bin/gym", line 4, in <module>
from nemo_gym.cli.main import main
PermissionError: [Errno 13] Permission denied:
'/opt/nemo-gym/nemo_gym/__init__.py'
The non-root probe added in NVIDIA-NeMo#2560 turns that into a build failure, so the
image currently does not build under NeMo CI on either amd64 or arm64.
Add --chown to the release-stage copy, matching the hermetic-stage copies
directly above it. Applying ownership during COPY costs no extra layer,
so the reason the later pass skips files still holds, and the image no
longer depends on the umask of whoever builds it.
Signed-off-by: Kajal Jain <kajalj@nvidia.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What does this PR do?
Adds
--chownto the release-stage source copy indocker/Dockerfile, matching the hermetic-stage copies directly above it.Why
The release stage copies the full checkout without
--chown, so the source keeps whatever ownership and mode the build context supplied. The ownership pass that follows covers directories only — deliberately, to avoid duplicating the source tree into another layer:find /opt/nemo-gym ... -type d -exec chown "${RUNTIME_UID}:${RUNTIME_GID}" {} +That leaves the files root-owned. Fine whenever the context is world-readable, which is true for the self-contained path where the
nemo-gymstage is populated byADDfrom GitHub — so this passes in our own CI.It does not hold for every builder. NeMo CI builds this image with
--build-context nemo-gym=.from a checkout made underumask 0007, so files arrive as mode0660owned by root. AfterUSER ${RUNTIME_UID}the interpreter cannot read them:The non-root probe added in #2560 turns that into a hard build failure. The image does not currently build under NeMo CI on either amd64 or arm64 — failing at
Dockerfile:214, identically on both.This went unnoticed because NeMo CI stopped building this Dockerfile on 2026-08-11 (an unrelated CI change repointed the Gym build at a different Dockerfile). Restoring that build is what surfaced it.
Why this fix
Applying ownership during
COPYcosts no extra layer, so the reason the later pass skips files still holds. The image stops depending on the umask of whoever builds it, which seems worth having regardless of who the builder is.An alternative would be fixing modes on the NeMo CI side before the build. That works too, but leaves the image sensitive to build-context permissions for anyone else building it. Happy to go that route instead if you'd prefer the Dockerfile stay as-is.
Testing
docker build --check -f docker/Dockerfile .— clean, no warningspre-commit run --files docker/Dockerfile— passmain, both arches fail)f8d4318b6, which predates fix(docker): support non-root runtime execution #2560, builds green on both arches — 63619803I have not yet run a full image build with this change applied — a NeMo CI run against this branch is the direct verification and I'd like to do that before this merges. Flagging it rather than implying more validation than I've done.