Skip to content

fix(docker): make source readable by the runtime UID - #2632

Open
kajalj22 wants to merge 1 commit into
NVIDIA-NeMo:mainfrom
kajalj22:kajalj/fix-nonroot-source-perms
Open

fix(docker): make source readable by the runtime UID#2632
kajalj22 wants to merge 1 commit into
NVIDIA-NeMo:mainfrom
kajalj22:kajalj/fix-nonroot-source-perms

Conversation

@kajalj22

Copy link
Copy Markdown
Contributor

What does this PR do?

Adds --chown to the release-stage source copy in docker/Dockerfile, matching the hermetic-stage copies directly above it.

Why

The release stage copies the full checkout without --chown, so the source keeps whatever ownership and mode the build context supplied. The ownership pass that follows covers directories only — deliberately, to avoid duplicating the source tree into another layer:

find /opt/nemo-gym ... -type d -exec chown "${RUNTIME_UID}:${RUNTIME_GID}" {} +

That leaves the files root-owned. Fine whenever the context is world-readable, which is true for the self-contained path where the nemo-gym stage is populated by ADD from GitHub — so this passes in our own CI.

It does not hold for every builder. NeMo CI builds this image with --build-context nemo-gym=. from a checkout made under umask 0007, so files arrive as mode 0660 owned by root. After USER ${RUNTIME_UID} the interpreter cannot read them:

File "/opt/nemo_gym_venv/bin/gym", line 4, in <module>
    from nemo_gym.cli.main import main
  ...
PermissionError: [Errno 13] Permission denied: '/opt/nemo-gym/nemo_gym/__init__.py'

The non-root probe added in #2560 turns that into a hard build failure. The image does not currently build under NeMo CI on either amd64 or arm64 — failing at Dockerfile:214, identically on both.

This went unnoticed because NeMo CI stopped building this Dockerfile on 2026-08-11 (an unrelated CI change repointed the Gym build at a different Dockerfile). Restoring that build is what surfaced it.

Why this fix

Applying ownership during COPY costs no extra layer, so the reason the later pass skips files still holds. The image stops depending on the umask of whoever builds it, which seems worth having regardless of who the builder is.

An alternative would be fixing modes on the NeMo CI side before the build. That works too, but leaves the image sensitive to build-context permissions for anyone else building it. Happy to go that route instead if you'd prefer the Dockerfile stay as-is.

Testing

I have not yet run a full image build with this change applied — a NeMo CI run against this branch is the direct verification and I'd like to do that before this merges. Flagging it rather than implying more validation than I've done.

The release stage copies the full Git checkout without --chown, so the
source keeps whatever ownership and mode the build context supplied. The
ownership pass that follows deliberately covers directories only, to
avoid duplicating the source tree into another layer:

    find /opt/nemo-gym ... -type d -exec chown ... {} +

That leaves the files themselves root-owned. It works whenever the build
context is world-readable, which is the case for the self-contained path
where the nemo-gym stage is populated by ADD from GitHub.

It does not hold for every builder. NeMo CI builds this image with
`--build-context nemo-gym=.` from a checkout made under `umask 0007`, so
the files arrive as mode 0660 owned by root. After `USER ${RUNTIME_UID}`
the interpreter can no longer read them:

    File "/opt/nemo_gym_venv/bin/gym", line 4, in <module>
        from nemo_gym.cli.main import main
    PermissionError: [Errno 13] Permission denied:
        '/opt/nemo-gym/nemo_gym/__init__.py'

The non-root probe added in NVIDIA-NeMo#2560 turns that into a build failure, so the
image currently does not build under NeMo CI on either amd64 or arm64.

Add --chown to the release-stage copy, matching the hermetic-stage copies
directly above it. Applying ownership during COPY costs no extra layer,
so the reason the later pass skips files still holds, and the image no
longer depends on the umask of whoever builds it.

Signed-off-by: Kajal Jain <kajalj@nvidia.com>
@kajalj22
kajalj22 requested a review from a team as a code owner August 20, 2026 05:29
@copy-pr-bot

copy-pr-bot Bot commented Aug 20, 2026

Copy link
Copy Markdown

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant