Unable to run nydus images from private registry(ecr) aws

[gane5hvarma]

Additional Information

Hi, I'm running nydus snapshotter in eks(k8s). Im facing a problem when nydus images are stored in private registry(ecr). Following are the logs of the pod. Normal docker images from ecr are working fine, the problem is with nydus image

Warning  Failed  13s               kubelet  Error: failed to create containerd container: mount callback failed on /var/lib/containerd/tmpmounts/containerd-mount1813997578: read /var/lib/containerd/tmpmounts/containerd-mount1813997578/etc/passwd: invalid argument
  Warning  Failed  13s               kubelet  Error: failed to create containerd container: mount callback failed on /var/lib/containerd/tmpmounts/containerd-mount2887762423: read /var/lib/containerd/tmpmounts/containerd-mount2887762423/etc/passwd: invalid argument
  Normal   Pulled  1s (x3 over 14s)  kubelet  Container image "089962644720.dkr.ecr.us-east-1.amazonaws.com/nydus-test:7b1afd3b8ef94e638570136cf88e0738-nydus" already present on machine
  Warning  Failed  1s                kubelet  Error: failed to create containerd container: mount callback failed on /var/lib/containerd/tmpmounts/containerd-mount3406305402: read /var/lib/containerd/tmpmounts/containerd-mount3406305402/etc/passwd: invalid argument

Version of nydus being used (nydusd --version)

Version of nydus-snapshotter being used (containerd-nydus-grpc --version)

Kernel information (uname -r)

_command result: x.xx.xxx-xxx.xxx.amzn2.x86_64

GNU/Linux Distribution, if applicable (cat /etc/os-release)

command result: cat /etc/os-release

containerd-nydus-grpc command line used, if applicable (ps aux | grep containerd-nydus-grpc)

containerd-nydus-grpc --nydusd /usr/local/bin/nydusd --nydusd-config /etc/nydus/config.json --root /var/lib/containerd-nydus --address /run/containerd-nydus/containerd-nydus-grpc.sock --log-level info --daemon-mode multiple --log-to-stdout

client command line used, if applicable (such as: nerdctl, docker, kubectl, ctr)

kubectl

Screenshots (if applicable)

Details about issue

[gane5hvarma]

@imeoer any ideas on how to resolve this ?

[imeoer]

@gane5hvarma Any error logs can be found on containerd-nydus-grpc process?

[gane5hvarma]

@imeoer i see this logs in nydus-snapshotter pod

[2024-11-22 12:02:28.947022 +00:00] WARN [storage/src/backend/mod.rs:91] Read from backend failed: Registry(Common("Invalid argument (os error 22)")), retry count 1
[2024-11-22 12:02:29.937703 +00:00] ERROR [error/src/error.rs:21] Error:
	"invalid auth config"
	at storage/src/backend/registry.rs:289
	note: enable `RUST_BACKTRACE=1` env to display a backtrace
[2024-11-22 12:02:29.937846 +00:00] ERROR [error/src/error.rs:21] Error:
	"failed to read metadata from backend, Registry(Common(\"Invalid argument (os error 22)\"))"
	at storage/src/meta/mod.rs:674
	note: enable `RUST_BACKTRACE=1` env to display a backtrace
[2024-11-22 12:02:29.937901 +00:00] INFO [storage/src/cache/cachedfile.rs:74] temporarily failed to get blob.meta, I/O error (os error 5)
[2024-11-22 12:02:29.940076 +00:00] ERROR [error/src/error.rs:21] Error:
	"invalid auth config"
	at storage/src/backend/registry.rs:289
	note: enable `RUST_BACKTRACE=1` env to display a backtrace
[2024-11-22 12:02:29.940129 +00:00] ERROR [error/src/error.rs:21] Error:
	"failed to read metadata from backend(compressor is none), Registry(Common(\"Invalid argument (os error 22)\"))"
	at storage/src/meta/mod.rs:659
	note: enable `RUST_BACKTRACE=1` env to display a backtrace
[2024-11-22 12:02:29.940164 +00:00] INFO [storage/src/cache/cachedfile.rs:74] temporarily failed to get blob.meta, I/O error (os error 5)
[2024-11-22 12:02:29.942024 +00:00] ERROR [error/src/error.rs:21] Error:
	"invalid auth config"
	at storage/src/backend/registry.rs:289
	note: enable `RUST_BACKTRACE=1` env to display a backtrace
[2024-11-22 12:02:29.942179 +00:00] ERROR [error/src/error.rs:21] Error:
	"failed to read metadata from backend, Registry(Common(\"Invalid argument (os error 22)\"))"
	at storage/src/meta/mod.rs:674
	note: enable `RUST_BACKTRACE=1` env to display a backtrace
[2024-11-22 12:02:29.942276 +00:00] INFO [storage/src/cache/cachedfile.rs:74] temporarily failed to get blob.meta, I/O error (os error 5)
[2024-11-22 12:02:29.943250 +00:00] ERROR [error/src/error.rs:21] Error:
	"invalid auth config"
	at storage/src/backend/registry.rs:289
	note: enable `RUST_BACKTRACE=1` env to display a backtrace
[2024-11-22 12:02:29.943298 +00:00] ERROR [error/src/error.rs:21] Error:
	"failed to read metadata from backend, Registry(Common(\"Invalid argument (os error 22)\"))"
	at storage/src/meta/mod.rs:674
	note: enable `RUST_BACKTRACE=1` env to display a backtrace
[2024-11-22 12:02:29.943313 +00:00] INFO [storage/src/cache/cachedfile.rs:74] temporarily failed to get blob.meta, I/O error (os error 5)
[2024-11-22 12:02:29.948041 +00:00] ERROR [error/src/error.rs:21] Error:
	"invalid auth config"
	at storage/src/backend/registry.rs:289
	note: enable `RUST_BACKTRACE=1` env to display a backtrace
[2024-11-22 12:02:29.948087 +00:00] ERROR [error/src/error.rs:21] Error:
	"failed to read metadata from backend, Registry(Common(\"Invalid argument (os error 22)\"))"
	at storage/src/meta/mod.rs:674
	note: enable `RUST_BACKTRACE=1` env to display a backtrace
[2024-11-22 12:02:29.948103 +00:00] INFO [storage/src/cache/cachedfile.rs:74] temporarily failed to get blob.meta, I/O error (os error 5)
[2024-11-22 12:02:36.338030 +00:00] WARN [storage/src/cache/cachedfile.rs:80] failed to get blob.meta
[2024-11-22 12:02:36.339083 +00:00] WARN [storage/src/cache/worker.rs:364] storage: failed to prefetch data from blob 07ff2aeacdf248f0a05ff014f1df2e915c67383a684d426d821e4906fa5e2a23, offset 14680064, size 2097152, Invalid argument (os error 22): storage/src/cache/cachedfile.rs:418, will try resend
[2024-11-22 12:02:36.339142 +00:00] WARN [storage/src/cache/worker.rs:364] storage: failed to prefetch data from blob 07ff2aeacdf248f0a05ff014f1df2e915c67383a684d426d821e4906fa5e2a23, offset 0, size 2097152, Invalid argument (os error 22): storage/src/cache/cachedfile.rs:418, will try resend

[gane5hvarma]

Im using nydus-snapshotter v0.13.0 version

[imeoer]

@gane5hvarma It looks like no auth info is provided in docker config (/root/.docker/config.json by default).

[gane5hvarma]

@imeoer i have followed this guide on how to enable auth
https://github.com/containerd/nydus-snapshotter/blob/main/docs/configure_nydus.md#use-serviceaccount
and im using https://github.com/dragonflyoss/helm-charts helm charts which creates clusterRole and clusterRoleBinding

[imeoer]

@gane5hvarma It seems needs some debugging for the code: https://github.com/containerd/nydus-snapshotter/blob/5f948e4498151b51c742d2ee0b3f7b96f86a26f7/pkg/auth/kubesecret.go#L162

[gane5hvarma]

Hi @imeoer. Is there a dockerfile for nydus snapshotter. I made some changes to nydus snapshotter and added some debugging code. I need a dockerfile to build docker image and test in eks

[imeoer]

Hi @gane5hvarma, a example Dockerfile in here

[gane5hvarma]

Thanks @imeoer. I was able to make changes and build the docker image.
@imeoer Is there a way to know if nydus-snapshotter has fetched all the layers of an image?.
The container starts immediately, but i wanted to know how much time it took to pull all the layers in the background

[imeoer]

Hi @gane5hvarma There is currently no clear way to obtain the complete prefetch time for the image, and typically users do not need to be concerned about it, but you can check the /var/lib/containerd/io.containerd.snapshotter.v1.nydus/cache (by default) directory to find the latest modification time of layer cache files.

[gane5hvarma]

Hi @imeoer im seeing this errors in nydus snapshotter. Any idea on how to resolve this?

[2024-12-10 11:25:04.454648 +00:00] ERROR [error/src/error.rs:21] Error:
	"start: 2291827125, end: 2291846821, addr: 2292187136"
	at storage/src/meta/mod.rs:771
	note: enable `RUST_BACKTRACE=1` env to display a backtrace
[2024-12-10 11:25:04.454681 +00:00] WARN [storage/src/cache/worker.rs:364] storage: failed to prefetch data from blob b959e792b4885087c5753b627814408bdf9a21430298887a992b11e80c1481bf, offset 2292187136, size 168708, Invalid argument (os error 22), will try resend
[2024-12-10 11:25:04.460892 +00:00] WARN [storage/src/cache/worker.rs:364] storage: failed to prefetch data from blob 229a556903eda3c1b8c930efc13ad01b3c4d25a3b234ee1a69e150abf2b8e98e, offset 0, size 96616, Invalid argument (os error 22): storage/src/meta/mod.rs:568, will try resend