pod CreateContainerError when deploy according INSTALL.md

[jokerwenxiao]

Bug report:

When I tested according to the steps in INSTALL.md, there was an error deploying the NGINX pod

pod events:

Events:
  Type     Reason     Age                From               Message
  ----     ------     ----               ----               -------
  Normal   Scheduled  43s                default-scheduler  Successfully assigned default/nginx to kind-control-plane
  Normal   Pulled     40s                kubelet            Successfully pulled image "ghcr.io/dragonflyoss/image-service/nginx:nydus-latest" in 1.470385199s (1.470388746s including waiting)
  Warning  Failed     19s                kubelet            Error: failed to create containerd container: mount callback failed on /var/lib/containerd/tmpmounts/containerd-mount1614983617: read /var/lib/containerd/tmpmounts/containerd-mount1614983617/etc/passwd: input/output error
  Normal   Pulling    18s (x2 over 41s)  kubelet            Pulling image "ghcr.io/dragonflyoss/image-service/nginx:nydus-latest"
  Normal   Pulled     17s                kubelet            Successfully pulled image "ghcr.io/dragonflyoss/image-service/nginx:nydus-latest" in 1.54053078s (1.540535229s including waiting)

kubelet log:

Sep 04 08:42:36 kind-control-plane kubelet[214]: I0904 08:42:36.186657     214 topology_manager.go:200] "Topology Admit Handler"
Sep 04 08:42:36 kind-control-plane kubelet[214]: I0904 08:42:36.324750     214 reconciler.go:238] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-57b29\" (UniqueName: \"kubernetes.io/projected/3e90bdfa-1f46-4f8c-ad64-e490c7a32095-kube-api-access-57b29\") pod \"nginx\" (UID: \"3e90bdfa-1f46-4f8c-ad64-e490c7a32095\") " pod="default/nginx"
Sep 04 08:43:00 kind-control-plane kubelet[214]: E0904 08:43:00.211022     214 remote_runtime.go:416] "CreateContainer in sandbox from runtime service failed" err="rpc error: code = Unknown desc = failed to create containerd container: mount callback failed on /var/lib/containerd/tmpmounts/containerd-mount1614983617: read /var/lib/containerd/tmpmounts/containerd-mount1614983617/etc/passwd: input/output error" podSandboxID="13d3d6d05234d011ee1fc0f6ac78473c2afa0b750bd66f7c08eb753659c4b67c"
Sep 04 08:43:00 kind-control-plane kubelet[214]: E0904 08:43:00.211143     214 kuberuntime_manager.go:919] container &Container{Name:nginx,Image:ghcr.io/dragonflyoss/image-service/nginx:nydus-latest,Command:[sh -c],Args:[tail -f /dev/null],WorkingDir:,Ports:[]ContainerPort{},Env:[]EnvVar{},Resources:ResourceRequirements{Limits:ResourceList{},Requests:ResourceList{},},VolumeMounts:[]VolumeMount{VolumeMount{Name:kube-api-access-57b29,ReadOnly:true,MountPath:/var/run/secrets/kubernetes.io/serviceaccount,SubPath:,MountPropagation:nil,SubPathExpr:,},},LivenessProbe:nil,ReadinessProbe:nil,Lifecycle:nil,TerminationMessagePath:/dev/termination-log,ImagePullPolicy:Always,SecurityContext:nil,Stdin:false,StdinOnce:false,TTY:false,EnvFrom:[]EnvFromSource{},TerminationMessagePolicy:File,VolumeDevices:[]VolumeDevice{},StartupProbe:nil,} start failed in pod nginx_default(3e90bdfa-1f46-4f8c-ad64-e490c7a32095): CreateContainerError: failed to create containerd container: mount callback failed on /var/lib/containerd/tmpmounts/containerd-mount1614983617: read /var/lib/containerd/tmpmounts/containerd-mount1614983617/etc/passwd: input/output error
Sep 04 08:43:00 kind-control-plane kubelet[214]: E0904 08:43:00.211181     214 pod_workers.go:965] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"nginx\" with CreateContainerError: \"failed to create containerd container: mount callback failed on /var/lib/containerd/tmpmounts/containerd-mount1614983617: read /var/lib/containerd/tmpmounts/containerd-mount1614983617/etc/passwd: input/output error\"" pod="default/nginx" podUID=3e90bdfa-1f46-4f8c-ad64-e490c7a32095
Sep 04 08:43:23 kind-control-plane kubelet[214]: E0904 08:43:23.203762     214 remote_runtime.go:416] "CreateContainer in sandbox from runtime service failed" err="rpc error: code = Unknown desc = failed to create containerd container: mount callback failed on /var/lib/containerd/tmpmounts/containerd-mount3881930381: read /var/lib/containerd/tmpmounts/containerd-mount3881930381/etc/passwd: input/output error" podSandboxID="13d3d6d05234d011ee1fc0f6ac78473c2afa0b750bd66f7c08eb753659c4b67c"
Sep 04 08:43:23 kind-control-plane kubelet[214]: E0904 08:43:23.203865     214 kuberuntime_manager.go:919] container &Container{Name:nginx,Image:ghcr.io/dragonflyoss/image-service/nginx:nydus-latest,Command:[sh -c],Args:[tail -f /dev/null],WorkingDir:,Ports:[]ContainerPort{},Env:[]EnvVar{},Resources:ResourceRequirements{Limits:ResourceList{},Requests:ResourceList{},},VolumeMounts:[]VolumeMount{VolumeMount{Name:kube-api-access-57b29,ReadOnly:true,MountPath:/var/run/secrets/kubernetes.io/serviceaccount,SubPath:,MountPropagation:nil,SubPathExpr:,},},LivenessProbe:nil,ReadinessProbe:nil,Lifecycle:nil,TerminationMessagePath:/dev/termination-log,ImagePullPolicy:Always,SecurityContext:nil,Stdin:false,StdinOnce:false,TTY:false,EnvFrom:[]EnvFromSource{},TerminationMessagePolicy:File,VolumeDevices:[]VolumeDevice{},StartupProbe:nil,} start failed in pod nginx_default(3e90bdfa-1f46-4f8c-ad64-e490c7a32095): CreateContainerError: failed to create containerd container: mount callback failed on /var/lib/containerd/tmpmounts/containerd-mount3881930381: read /var/lib/containerd/tmpmounts/containerd-mount3881930381/etc/passwd: input/output error
Sep 04 08:43:23 kind-control-plane kubelet[214]: E0904 08:43:23.203899     214 pod_workers.go:965] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"nginx\" with CreateContainerError: \"failed to create containerd container: mount callback failed on /var/lib/containerd/tmpmounts/containerd-mount3881930381: read /var/lib/containerd/tmpmounts/containerd-mount3881930381/etc/passwd: input/output error\"" pod="default/nginx" podUID=3e90bdfa-1f46-4f8c-ad64-e490c7a32095

nydus-snapshotter pod log:

time="2024-09-04T08:42:15.752400018Z" level=info msg="[Cleanup] orphan directories [/var/lib/containerd-nydus/snapshots/439]"
time="2024-09-04T08:42:38.152455616Z" level=info msg="[Mounts] snapshot k8s.io/470/13d3d6d05234d011ee1fc0f6ac78473c2afa0b750bd66f7c08eb753659c4b67c ID 468 Kind Active"
time="2024-09-04T08:42:39.704748418Z" level=info msg="Prepares active snapshot k8s.io/471/0fa090b5c0091acff6f6955b49a3c19efa689651f1ece985b3adb304a12da12f, nydusd should start afterwards" key=k8s.io/471/0fa090b5c0091acff6f6955b49a3c19efa689651f1ece985b3adb304a12da12f parent="k8s.io/11/sha256:f7a1174747879fefb6b2dbea73b25fe692b41825ef22f70ee21b1b301fb657ff"
time="2024-09-04T08:42:39.704785089Z" level=info msg="Nydus remote snapshot 9 is ready"
time="2024-09-04T08:42:39.704790490Z" level=info msg="Nydus remote snapshot 9 is ready"
time="2024-09-04T08:42:39.704801601Z" level=info msg="remote mount options [workdir=/var/lib/containerd-nydus/snapshots/469/work upperdir=/var/lib/containerd-nydus/snapshots/469/fs lowerdir=/var/lib/containerd-nydus/snapshots/9/mnt]"
time="2024-09-04T08:42:39.705652387Z" level=info msg="[Mounts] snapshot k8s.io/471/0fa090b5c0091acff6f6955b49a3c19efa689651f1ece985b3adb304a12da12f ID 469 Kind Active"
time="2024-09-04T08:42:39.705669600Z" level=info msg="Nydus remote snapshot 9 is ready"
time="2024-09-04T08:42:39.705685702Z" level=info msg="remote mount options [workdir=/var/lib/containerd-nydus/snapshots/469/work upperdir=/var/lib/containerd-nydus/snapshots/469/fs lowerdir=/var/lib/containerd-nydus/snapshots/9/mnt]"
[2024-09-04 08:42:41.706689 +00:00] WARN [storage/src/cache/state/blob_state_map.rs:120] Waiting for backend IO expires. chunk index 5, compressed offset 7377
[2024-09-04 08:42:43.707298 +00:00] WARN [storage/src/cache/state/blob_state_map.rs:120] Waiting for backend IO expires. chunk index 6, compressed offset 7959
[2024-09-04 08:42:48.707620 +00:00] WARN [storage/src/backend/mod.rs:91] Read from backend failed: Registry(Request(Common(reqwest::Error { kind: Request, url: Url { scheme: "https", cannot_be_a_base: false, username: "", password: None, host: Some(Domain("ghcr.io")), port: None, path: "/v2/dragonflyoss/image-service/nginx/blobs/sha256:a877ac8e60e55738ac9a27ed567b13843d46e4a71ce044811d0e30de06b68c94", query: None, fragment: None }, source: TimedOut }))), retry count 2
[2024-09-04 08:42:54.208436 +00:00] WARN [storage/src/backend/mod.rs:91] Read from backend failed: Registry(Request(Common(reqwest::Error { kind: Request, url: Url { scheme: "https", cannot_be_a_base: false, username: "", password: None, host: Some(Domain("ghcr.io")), port: None, path: "/v2/dragonflyoss/image-service/nginx/blobs/sha256:a877ac8e60e55738ac9a27ed567b13843d46e4a71ce044811d0e30de06b68c94", query: None, fragment: None }, source: TimedOut }))), retry count 1
[2024-09-04 08:43:00.208952 +00:00] ERROR [error/src/error.rs:21] Error:
        Registry(Request(Common(reqwest::Error { kind: Request, url: Url { scheme: "https", cannot_be_a_base: false, username: "", password: None, host: Some(Domain("ghcr.io")), port: None, path: "/v2/dragonflyoss/image-service/nginx/blobs/sha256:a877ac8e60e55738ac9a27ed567b13843d46e4a71ce044811d0e30de06b68c94", query: None, fragment: None }, source: TimedOut })))
        at storage/src/cache/mod.rs:207
        note: enable `RUST_BACKTRACE=1` env to display a backtrace
[2024-09-04 08:43:00.208997 +00:00] ERROR [/cargo/registry/src/github.com-1ecc6299db9ec823/fuse-backend-rs-0.9.6/src/api/server/sync_io.rs:1180] fuse: reply error header OutHeader { len: 16, error: -5, unique: 4616 }, error Os { code: 5, kind: Uncategorized, message: "I/O error" }
time="2024-09-04T08:43:00.425075289Z" level=info msg="[Cleanup] orphan directories [/var/lib/containerd-nydus/snapshots/469]"
time="2024-09-04T08:43:02.692829844Z" level=info msg="Prepares active snapshot k8s.io/472/3a124f67f76a572e18dc780272f44896bb901c998bf5ccf2aa8c21ea5291c118, nydusd should start afterwards" key=k8s.io/472/3a124f67f76a572e18dc780272f44896bb901c998bf5ccf2aa8c21ea5291c118 parent="k8s.io/11/sha256:f7a1174747879fefb6b2dbea73b25fe692b41825ef22f70ee21b1b301fb657ff"
time="2024-09-04T08:43:02.692879661Z" level=info msg="Nydus remote snapshot 9 is ready"
time="2024-09-04T08:43:02.692884942Z" level=info msg="Nydus remote snapshot 9 is ready"
time="2024-09-04T08:43:02.692896324Z" level=info msg="remote mount options [workdir=/var/lib/containerd-nydus/snapshots/470/work upperdir=/var/lib/containerd-nydus/snapshots/470/fs lowerdir=/var/lib/containerd-nydus/snapshots/9/mnt]"
time="2024-09-04T08:43:02.695391961Z" level=info msg="[Mounts] snapshot k8s.io/472/3a124f67f76a572e18dc780272f44896bb901c998bf5ccf2aa8c21ea5291c118 ID 470 Kind Active"
time="2024-09-04T08:43:02.695420046Z" level=info msg="Nydus remote snapshot 9 is ready"
time="2024-09-04T08:43:02.695529268Z" level=info msg="remote mount options [workdir=/var/lib/containerd-nydus/snapshots/470/work upperdir=/var/lib/containerd-nydus/snapshots/470/fs lowerdir=/var/lib/containerd-nydus/snapshots/9/mnt]"
[2024-09-04 08:43:04.698170 +00:00] WARN [storage/src/cache/state/blob_state_map.rs:120] Waiting for backend IO expires. chunk index 5, compressed offset 7377
[2024-09-04 08:43:06.698608 +00:00] WARN [storage/src/cache/state/blob_state_map.rs:120] Waiting for backend IO expires. chunk index 6, compressed offset 7959
[2024-09-04 08:43:11.698944 +00:00] WARN [storage/src/backend/mod.rs:91] Read from backend failed: Registry(Request(Common(reqwest::Error { kind: Request, url: Url { scheme: "https", cannot_be_a_base: false, username: "", password: None, host: Some(Domain("ghcr.io")), port: None, path: "/v2/dragonflyoss/image-service/nginx/blobs/sha256:a877ac8e60e55738ac9a27ed567b13843d46e4a71ce044811d0e30de06b68c94", query: None, fragment: None }, source: TimedOut }))), retry count 2
[2024-09-04 08:43:17.199828 +00:00] WARN [storage/src/backend/mod.rs:91] Read from backend failed: Registry(Request(Common(reqwest::Error { kind: Request, url: Url { scheme: "https", cannot_be_a_base: false, username: "", password: None, host: Some(Domain("ghcr.io")), port: None, path: "/v2/dragonflyoss/image-service/nginx/blobs/sha256:a877ac8e60e55738ac9a27ed567b13843d46e4a71ce044811d0e30de06b68c94", query: None, fragment: None }, source: TimedOut }))), retry count 1
[2024-09-04 08:43:23.200441 +00:00] ERROR [error/src/error.rs:21] Error:
        Registry(Request(Common(reqwest::Error { kind: Request, url: Url { scheme: "https", cannot_be_a_base: false, username: "", password: None, host: Some(Domain("ghcr.io")), port: None, path: "/v2/dragonflyoss/image-service/nginx/blobs/sha256:a877ac8e60e55738ac9a27ed567b13843d46e4a71ce044811d0e30de06b68c94", query: None, fragment: None }, source: TimedOut })))
        at storage/src/cache/mod.rs:207
        note: enable `RUST_BACKTRACE=1` env to display a backtrace
[2024-09-04 08:43:23.200507 +00:00] ERROR [/cargo/registry/src/github.com-1ecc6299db9ec823/fuse-backend-rs-0.9.6/src/api/server/sync_io.rs:1180] fuse: reply error header OutHeader { len: 16, error: -5, unique: 4626 }, error Os { code: 5, kind: Uncategorized, message: "I/O error" }
time="2024-09-04T08:43:23.234780304Z" level=info msg="[Cleanup] orphan directories [/var/lib/containerd-nydus/snapshots/470]"
time="2024-09-04T08:43:26.150099852Z" level=info msg="Prepares active snapshot k8s.io/473/b0de897205298e4f4cef3730e63f0416440f794cd0453e604c00f0a6b3d3bf53, nydusd should start afterwards" key=k8s.io/473/b0de897205298e4f4cef3730e63f0416440f794cd0453e604c00f0a6b3d3bf53 parent="k8s.io/11/sha256:f7a1174747879fefb6b2dbea73b25fe692b41825ef22f70ee21b1b301fb657ff"
time="2024-09-04T08:43:26.150243962Z" level=info msg="Nydus remote snapshot 9 is ready"
time="2024-09-04T08:43:26.150265785Z" level=info msg="Nydus remote snapshot 9 is ready"
time="2024-09-04T08:43:26.150293529Z" level=info msg="remote mount options [workdir=/var/lib/containerd-nydus/snapshots/471/work upperdir=/var/lib/containerd-nydus/snapshots/471/fs lowerdir=/var/lib/containerd-nydus/snapshots/9/mnt]"
time="2024-09-04T08:43:26.151552388Z" level=info msg="[Mounts] snapshot k8s.io/473/b0de897205298e4f4cef3730e63f0416440f794cd0453e604c00f0a6b3d3bf53 ID 471 Kind Active"
time="2024-09-04T08:43:26.151618537Z" level=info msg="Nydus remote snapshot 9 is ready"
time="2024-09-04T08:43:26.151743851Z" level=info msg="remote mount options [workdir=/var/lib/containerd-nydus/snapshots/471/work upperdir=/var/lib/containerd-nydus/snapshots/471/fs lowerdir=/var/lib/containerd-nydus/snapshots/9/mnt]"
[2024-09-04 08:43:28.153610 +00:00] WARN [storage/src/cache/state/blob_state_map.rs:120] Waiting for backend IO expires. chunk index 5, compressed offset 7377
[2024-09-04 08:43:30.154184 +00:00] WARN [storage/src/cache/state/blob_state_map.rs:120] Waiting for backend IO expires. chunk index 6, compressed offset 7959
[2024-09-04 08:43:35.154426 +00:00] WARN [storage/src/backend/mod.rs:91] Read from backend failed: Registry(Request(Common(reqwest::Error { kind: Request, url: Url { scheme: "https", cannot_be_a_base: false, username: "", password: None, host: Some(Domain("ghcr.io")), port: None, path: "/v2/dragonflyoss/image-service/nginx/blobs/sha256:a877ac8e60e55738ac9a27ed567b13843d46e4a71ce044811d0e30de06b68c94", query: None, fragment: None }, source: TimedOut }))), retry count 2
[2024-09-04 08:43:40.655234 +00:00] WARN [storage/src/backend/mod.rs:91] Read from backend failed: Registry(Request(Common(reqwest::Error { kind: Request, url: Url { scheme: "https", cannot_be_a_base: false, username: "", password: None, host: Some(Domain("ghcr.io")), port: None, path: "/v2/dragonflyoss/image-service/nginx/blobs/sha256:a877ac8e60e55738ac9a27ed567b13843d46e4a71ce044811d0e30de06b68c94", query: None, fragment: None }, source: TimedOut }))), retry count 1
[2024-09-04 08:43:46.656648 +00:00] ERROR [error/src/error.rs:21] Error:
        Registry(Request(Common(reqwest::Error { kind: Request, url: Url { scheme: "https", cannot_be_a_base: false, username: "", password: None, host: Some(Domain("ghcr.io")), port: None, path: "/v2/dragonflyoss/image-service/nginx/blobs/sha256:a877ac8e60e55738ac9a27ed567b13843d46e4a71ce044811d0e30de06b68c94", query: None, fragment: None }, source: TimedOut })))
        at storage/src/cache/mod.rs:207
        note: enable `RUST_BACKTRACE=1` env to display a backtrace
[2024-09-04 08:43:46.656693 +00:00] ERROR [/cargo/registry/src/github.com-1ecc6299db9ec823/fuse-backend-rs-0.9.6/src/api/server/sync_io.rs:1180] fuse: reply error header OutHeader { len: 16, error: -5, unique: 4636 }, error Os { code: 5, kind: Uncategorized, message: "I/O error" }

configs:

kind config:

kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
networking:
  ipFamily: dual
nodes:
  - role: control-plane
    image: kindest/node:v1.23.17
    extraPortMappings:
      - containerPort: 4001
        hostPort: 4001
        protocol: TCP
    extraMounts:
      - hostPath: ./config.toml
        containerPath: /etc/containerd/config.toml
      - hostPath: /tmp/artifact
        containerPath: /tmp/artifact
      - hostPath: /dev/fuse
        containerPath: /dev/fuse

containerd config:

# explicitly use v2 config format
version = 2

[debug]
  level = "debug"

[plugins."io.containerd.grpc.v1.cri".containerd]
  # save disk space when using a single snapshotter
  discard_unpacked_layers = false
  # explicitly use default snapshotter so we can sed it in entrypoint
  snapshotter = "nydus"
  # explicit default here, as we're configuring it below
  default_runtime_name = "runc"
disable_snapshot_annotations = false
[plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runc]
  # set default runtime handler to v2, which has a per-pod shim
  runtime_type = "io.containerd.runc.v2"

# Setup a runtime with the magic name ("test-handler") used for Kubernetes
# runtime class tests ...
[plugins."io.containerd.grpc.v1.cri".containerd.runtimes.test-handler]
  runtime_type = "io.containerd.runc.v2"

[plugins."io.containerd.grpc.v1.cri"]
  # use fixed sandbox image
  sandbox_image = "registry.k8s.io/pause:3.5"
  # allow hugepages controller to be missing
  # see https://github.com/containerd/cri/pull/1501
  tolerate_missing_hugepages_controller = true
  # restrict_oom_score_adj needs to be true when running inside UserNS (rootless)
  restrict_oom_score_adj = false

# Mirrors use d7y.io instead of docker.io.
# when using docker.io to generate an endpoint, https://index.docker.io will be returned by default.
# https://github.com/containerd/containerd/blob/main/pkg/cri/server/image_pull.go#L456
# https://github.com/containerd/containerd/blob/main/remotes/docker/resolver.go#L121
[plugins."io.containerd.grpc.v1.cri".registry.mirrors."d7y.io"]
  endpoint = ["http://127.0.0.1:65001"]
[proxy_plugins]
  [proxy_plugins.nydus]
    type = "snapshot"
    address = "/run/containerd-nydus/containerd-nydus-grpc.sock"

nydus helm config:

name: nydus-snapshotter
pullPolicy: Always
hostNetwork: true
resources:
  requests:
    cpu: "0"
    memory: "0"
  limits:
    cpu: "1"
    memory: "1Gi"

dragonfly:
  enable: true
  mirrorConfig:
  - host: "http://127.0.0.1:65001"
    auth_through: false
    headers:
      "X-Dragonfly-Registry": "https://ghcr.io"
    ping_url: "http://127.0.0.1:40901/server/ping"

containerRuntime:
  initContainerImage: ghcr.io/liubin/toml-cli:v0.0.7
  containerd:
    enable: true
    configFile: "/etc/containerd/config.toml"

Environment:

• Dragonfly version: v2.1.40
• OS: Ubuntu 22.04 LTS（vm）
• Kernel (e.g. uname -a): 6.8.0-40-generic（vm）