Private Auth for build fails if it's under a subpath #3748

zk-kb4 · 2024-12-10T15:05:11Z

Description

If you have the following path docker.domain.com/path/image the auth will fail for the build. However pulling the image with nerdctl pull docker.domain.com/path/image auths correctly and does not fail to pull the image. This is using Gitlab's container registry.

This happens on lima and on our Gitlab Runners.

Steps to reproduce the issue

Using the following dockerfile it will fail.

FROM docker.domain.com/path/image

CMD ['/usr/local/bin/binary']

Even though I've done nerdctl login docker.domain.com and successfully can run nerdctl pull docker.domain.com/path/image without issue. When it's in the dockerfile it fails to auth and pull the image for the build.

Describe the results you received and expected

Error received when attempting to build,

DEBU[0000] stateDir: /run/user/502/containerd-rootless
DEBU[0000] RootlessKit detach-netns mode: true
DEBU[0000] rootless parent main: executing "/usr/bin/nsenter" with [-r/ -w/home/zachk.linux --preserve-credentials -m -U -t 941 -F nerdctl build . --debug]
DEBU[0000] Choosing the buildkit host "unix:///run/user/502/buildkit-default/buildkitd.sock", candidates=[unix:///run/user/502/buildkit-default/buildkitd.sock unix:///run/user/502/buildkit/buildkitd.sock]
DEBU[0000] Chosen buildkit host "unix:///run/user/502/buildkit-default/buildkitd.sock"
DEBU[0000] worker labels: map[org.mobyproject.buildkit.worker.containerd.namespace:default org.mobyproject.buildkit.worker.containerd.uuid:13a80611-b828-4367-ae87-ecbb199fd05f org.mobyproject.buildkit.worker.executor:containerd org.mobyproject.buildkit.worker.hostname:lima-default org.mobyproject.buildkit.worker.network:cni org.mobyproject.buildkit.worker.selinux.enabled:false org.mobyproject.buildkit.worker.snapshotter:overlayfs]
DEBU[0000] running /usr/local/bin/buildctl [--addr=unix:///run/user/502/buildkit-default/buildkitd.sock build --progress=auto --frontend=dockerfile.v0 --local=context=. --output=type=image,unpack=true,dangling-name-prefix=<none> --local=dockerfile=/home/zachk.linux --opt=filename=Dockerfile]
[+] Building 0.5s (2/2) FINISHED
 => [internal] load build definition from Dockerfile                                             0.0s
 => => transferring dockerfile: 84B                                                              0.0s
 => ERROR [internal] load metadata for docker.domain.com/path/imagelatest          0.5s
------
 > [internal] load metadata for docker.domain.com/path/image:latest:
------
Dockerfile:1
--------------------
   1 | >>> FROM docker.domain.com/path/image
   2 |
--------------------
error: failed to solve: docker.domain.com/path/image: failed to resolve source metadata for docker.domain.com/path/image:latest: failed to authorize: failed to fetch anonymous token: unexpected status from GET request to https://gitlab.domain.com/jwt/auth?scope=repository%3Apath%2Fimage%3Apull&service=container_registry: 403 Forbidden

Image successfully pulls using nerdctl pull

nerdctl pull docker.domain.com/path/image
docker.domain.com/path/image:latest:                                 resolved       |++++++++++++++++++++++++++++++++++++++|
index-sha256:f30408c22ef8dc00df9458d5d83b28763d668ba29a9d4e78d1756bd9b2704a17:    exists         |++++++++++++++++++++++++++++++++++++++|
manifest-sha256:00bff3ec3457f3593b53879efb824830e377d92ea8409a37972a405b4f8cd632: exists         |++++++++++++++++++++++++++++++++++++++|
config-sha256:db1f9bd6e4ae0651d9345382ed9a7e1a63cefdcb2fda2351be02e3321baa83d7:   exists         |++++++++++++++++++++++++++++++++++++++|
layer-sha256:6e98b7393f17472b51c88b40af3813abcb463565f15f5d4c06cb7aa934328c8f:    exists         |++++++++++++++++++++++++++++++++++++++|
layer-sha256:360e8cbf2e62e25c22b7aa5d7fba3e0c14c23a855c8a576df991d4743b30d34e:    exists         |++++++++++++++++++++++++++++++++++++++|
elapsed: 1.8 s                                                                    total:   0.0 B (0.0 B/s)

Docker config,

{
	"auths": {
		"docker.domain.com:443": {
			"auth": "...."
		}
	}
}

What version of nerdctl are you using?

nerdctl version 2.0.1
containerd github.com/containerd/containerd/v2 v2.0.0 207ad711eabd375a01713109a8a197d197ff6542
buildctl github.com/moby/buildkit v0.18.0 95d190ef4f18b57c717eaad703b67cb2be781ebb

Are you using a variant of nerdctl? (e.g., Rancher Desktop)

None

Host information

Client:
 Namespace:	default
 Debug Mode:	false

Server:
 Server Version: v2.0.0
 Storage Driver: overlayfs
 Logging Driver: json-file
 Cgroup Driver: systemd
 Cgroup Version: 2
 Plugins:
  Log:     fluentd journald json-file none syslog
  Storage: native overlayfs fuse-overlayfs stargz
 Security Options:
  apparmor
  seccomp
   Profile: builtin
  cgroupns
  rootless
 Kernel Version:   6.11.0-9-generic
 Operating System: Ubuntu 24.10
 OSType:           linux
 Architecture:     aarch64
 CPUs:             4
 Total Memory:     3.813GiB
 Name:             lima-default
 ID:               13a80611-b828-4367-ae87-ecbb199fd05f

WARNING: bridge-nf-call-iptables is disabled
WARNING: bridge-nf-call-ip6tables is disabled

The text was updated successfully, but these errors were encountered:

apostasie · 2024-12-11T22:08:54Z

@AkihiroSuda will take this for investigation (in case it is a regression).

apostasie · 2024-12-12T01:21:01Z

@zk-kb4 I can't reproduce - it works for me.

What would be really useful to further diagnose this:
Can you try against an image on ghcr.io? if it still does not work, then something is different with your install - if on the other hand it does work, then this might be a gitlab issue (thinking specifically about #1675)

zk-kb4 · 2024-12-12T16:10:19Z

@apostasie I don't have any private images on ghcr.io to test unfortunately.

apostasie · 2024-12-13T03:02:02Z

@apostasie I don't have any private images on ghcr.io to test unfortunately.

It should be relatively easy to do.

go to https://github.com/settings/apps
then "Personal Access Tokens", then "Tokens (classic)", then "Generate new token (classic)"
select scope "Write packages", give it a "note" (eg: name), then "Generate token"
copy the token

Now, login with nerdctl:
nerdctl login ghcr.io
Use your github username and the token for password.

Then:

nerdctl pull busybox
nerdctl tag busybox ghcr.io/zk-kb4/test
nerdctl push ghcr.io/zk-kb4/test

The created image ghcr.io/zk-kb4/test should be private by default.

zk-kb4 · 2024-12-13T14:19:25Z

This failed as well.

zachk@lima-default:~$ nerdctl build -f Dockerfile.2 .
[+] Building 0.8s (2/2) FINISHED
 => [internal] load build definition from Dockerfile.2                                                                                                                      0.0s
 => => transferring dockerfile: 86B                                                                                                                                         0.0s
 => ERROR [internal] load metadata for ghcr.io/zk-kb4/test-nerd:latest                                                                                                      0.6s
------
 > [internal] load metadata for ghcr.io/zk-kb4/test-nerd:latest:
------
Dockerfile.2:1
--------------------
   1 | >>> FROM ghcr.io/zk-kb4/test-nerd
   2 |
   3 |     CMD ['busybox']
--------------------
error: failed to solve: ghcr.io/zk-kb4/test-nerd: failed to resolve source metadata for ghcr.io/zk-kb4/test-nerd:latest: failed to authorize: failed to fetch anonymous token: unexpected status from GET request to https://ghcr.io/token?scope=repository%3Azk-kb4%2Ftest-nerd%3Apull&service=ghcr.io: 401 Unauthorized

docker config,

{
	"auths": {
		"ghcr.io:443": {
			"auth": "...."
		}
	}

apostasie · 2024-12-13T16:55:03Z

Ok.

@zk-kb4 could you help with the following:

how did you install buildkit on that machine?
can you copy the output of nerdctl version
the output of buildctl --version

zk-kb4 · 2024-12-16T17:15:37Z

Ok.

@zk-kb4 could you help with the following:

how did you install buildkit on that machine?

can you copy the output of nerdctl version

the output of buildctl --version

It was already installed in lima

Client:
 Version:	v2.0.1
 OS/Arch:	linux/arm64
 Git commit:	47f31ff2c1615c1accb85c1ce4e7882ad739102f
 buildctl:
  Version:	v0.18.0
  GitCommit:	95d190ef4f18b57c717eaad703b67cb2be781ebb

Server:
 containerd:
  Version:	v2.0.0
  GitCommit:	207ad711eabd375a01713109a8a197d197ff6542
 runc:
  Version:	1.2.2
  GitCommit:	v1.2.2-0-g7cb36325

buildctl github.com/moby/buildkit v0.18.0 95d190ef4f18b57c717eaad703b67cb2be781ebb

apostasie · 2024-12-16T18:09:36Z

Ok.
@zk-kb4 could you help with the following:

how did you install buildkit on that machine?

can you copy the output of nerdctl version

the output of buildctl --version

It was already installed in lima
Client:
 Version:	v2.0.1
 OS/Arch:	linux/arm64
 Git commit:	47f31ff2c1615c1accb85c1ce4e7882ad739102f
 buildctl:
  Version:	v0.18.0
  GitCommit:	95d190ef4f18b57c717eaad703b67cb2be781ebb

Server:
 containerd:
  Version:	v2.0.0
  GitCommit:	207ad711eabd375a01713109a8a197d197ff6542
 runc:
  Version:	1.2.2
  GitCommit:	v1.2.2-0-g7cb36325
buildctl github.com/moby/buildkit v0.18.0 95d190ef4f18b57c717eaad703b67cb2be781ebb

Thanks.

Can you share which template / how you installed the lima instance?

apostasie · 2024-12-16T21:01:42Z

Scratch previous comment.

I managed to reproduce.

@AkihiroSuda let's mark this confirmed.

apostasie · 2024-12-16T21:20:24Z

@zk-kb4 right now, you should be able to workaround the problem by using the registry default port in your FROM:

FROM docker.domain.com:443/path/image

I will have a look at buildctl credentials resolution mechanism and see if the problem is here or over there.

apostasie · 2024-12-16T22:01:23Z

@zk-kb4 Confirming that buildctl resolution mechanism does not have logic to deal with domain.com vs. domain.com:443 and treat them differently.

Just opened a PR in #3766 to improve compatibility here.

zk-kb4 added the kind/unconfirmed-bug-claim Unconfirmed bug claim label Dec 10, 2024

AkihiroSuda added the area/login authentification/ login label Dec 10, 2024

apostasie linked a pull request Dec 16, 2024 that will close this issue

When the port is 443, also save credentials without port #3766

Open

AkihiroSuda added bug Something isn't working and removed kind/unconfirmed-bug-claim Unconfirmed bug claim labels Dec 17, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Private Auth for build fails if it's under a subpath #3748

Private Auth for build fails if it's under a subpath #3748

zk-kb4 commented Dec 10, 2024

apostasie commented Dec 11, 2024

apostasie commented Dec 12, 2024 •

edited

Loading

zk-kb4 commented Dec 12, 2024

apostasie commented Dec 13, 2024

zk-kb4 commented Dec 13, 2024 •

edited

Loading

apostasie commented Dec 13, 2024

zk-kb4 commented Dec 16, 2024

apostasie commented Dec 16, 2024

apostasie commented Dec 16, 2024 •

edited

Loading

apostasie commented Dec 16, 2024

apostasie commented Dec 16, 2024

Private Auth for build fails if it's under a subpath #3748

Private Auth for build fails if it's under a subpath #3748

Comments

zk-kb4 commented Dec 10, 2024

Description

Steps to reproduce the issue

Describe the results you received and expected

What version of nerdctl are you using?

Are you using a variant of nerdctl? (e.g., Rancher Desktop)

Host information

apostasie commented Dec 11, 2024

apostasie commented Dec 12, 2024 • edited Loading

zk-kb4 commented Dec 12, 2024

apostasie commented Dec 13, 2024

zk-kb4 commented Dec 13, 2024 • edited Loading

apostasie commented Dec 13, 2024

zk-kb4 commented Dec 16, 2024

apostasie commented Dec 16, 2024

apostasie commented Dec 16, 2024 • edited Loading

apostasie commented Dec 16, 2024

apostasie commented Dec 16, 2024

apostasie commented Dec 12, 2024 •

edited

Loading

zk-kb4 commented Dec 13, 2024 •

edited

Loading

apostasie commented Dec 16, 2024 •

edited

Loading