nerdctl in containerd, `unable to start container process: error during container init: error mounting "/var/lib/nerdctl/NNN/containers/default/NNN/resolv.conf" to rootfs at "/etc/resolv.conf": ... no such file or directory: unknown

[nicolasespiau]

Hey folks,

I need some help with an issue running nerdctl commands inside a gitlab runner deployed on kubernetes with containerd runtime.

From the beginning I only have one FATAL error which is level=fatal msg="no such file or directory" when running a nerdctl compose up but also nerd run hello-world

Thanks to @AkihiroSuda's guidance I've entered the runner pod and tried some debug-full command:

nerdctl --debug-full run hello-world

Which gave me:

DEBU[0000] failed to run [aa-exec -p nerdctl-default -- true]: "[38] aa-exec: ERROR: profile 'nerdctl-default' does not exist\n"  error="exit status 1"
DEBU[0000] verification process skipped                 
DEBU[0000] final cOpts is [0xb60420 0xfa47c0 0xb60840 0xb605a0 0xb602a0 0xfa5e40 0xfa71c0 0xb60d80] 
FATA[0000] failed to create shim task: OCI runtime create failed: runc create failed: unable to start container process: error during container init: error mounting "/var/lib/nerdctl/1935db59/containers/default/767aa2c2a216d8fce638c59234f87c251f88c88311b3f82d5bcf4dc37fdf0dbb/resolv.conf" to rootfs at "/etc/resolv.conf": stat /var/lib/nerdctl/1935db59/containers/default/767aa2c2a216d8fce638c59234f87c251f88c88311b3f82d5bcf4dc37fdf0dbb/resolv.conf: no such file or directory: unknown

I get that DEBU message can be ignored.

But the thing is, focusing on the FATA message, if I perform ls or stat myself I can find the supposed missing file:

root@runner-btbzx--project-25793245-concurrent-07qhsp:/# stat /var/lib/nerdctl/1935db59/containers/default/767aa2c2a216d8fce638c59234f87c251f88c88311b3f82d5bcf4dc37fdf0dbb/resolv.conf
  File: /var/lib/nerdctl/1935db59/containers/default/767aa2c2a216d8fce638c59234f87c251f88c88311b3f82d5bcf4dc37fdf0dbb/resolv.conf
  Size: 120             Blocks: 8          IO Block: 4096   regular file
Device: 10301h/66305d   Inode: 5133481     Links: 1
Access: (0644/-rw-r--r--)  Uid: (    0/    root)   Gid: (    0/    root)
Access: 2022-07-16 21:59:50.226552351 +0000
Modify: 2022-07-16 21:59:50.226552351 +0000
Change: 2022-07-16 21:59:50.226552351 +0000
 Birth: 2022-07-16 21:59:50.226552351 +0000

root@runner-btbzx--project-25793245-concurrent-07qhsp:/# ls -l /var/lib/nerdctl/1935db59/containers/default/767aa2c2a216d8fce638c59234f87c251f88c88311b3f82d5bcf4dc37fdf0dbb
total 8
-rw-r--r-- 1 root root  13 Jul 16 21:59 hostname
-rw-r--r-- 1 root root 120 Jul 16 21:59 resolv.conf

So why is the resolv.conf considered missing while it's actually here?

Here is my nerdctl info:

Client:
 Namespace:     default
 Debug Mode:    false

Server:
 Server Version: v1.6.6
 Storage Driver: overlayfs
 Logging Driver: json-file
 Cgroup Driver: cgroupfs
 Cgroup Version: 1
 Plugins:
  Log: fluentd journald json-file
  Storage: aufs native overlayfs
 Security Options:
  apparmor
  seccomp
   Profile: default
 Kernel Version: 5.13.0-1017-aws
 Operating System: Ubuntu 22.04 LTS
 OSType: linux
 Architecture: x86_64
 CPUs: 2
 Total Memory: 7.687GiB
 Name: runner-btbzx--project-25793245-concurrent-07qhsp
 ID: d4341431-1b20-4fc6-b55e-89dd1d3edd57

WARNING: IPv4 forwarding is disabled

[nicolasespiau]

I forgot to tell that I'm able to cat the missing resolv.conf and see the content which is the same as the node.

[austinarbor-wk]

@nicolasespiau did you ever figure this out? seeing this problem now

[nicolasespiau]

I'm sorry I don't remember what I did, my project was abandoned, I have no traces of what's been done

[dongzhiwei-git]

stat /var/lib/nerdctl/1935db59/containers/k8s.io/13cb58e1c9870196caa3450eea5553be5bdb469498c76bfefe95e82a08d12f0c/resolv.conf: no such file or directory: unknown

[dongzhiwei-git]

@austinarbor-wk

[dongzhiwei-git]

first：docker run -it --rm -v /var/lib/containerd:/var/lib/containerd --net=host --privileged -v /run/containerd/:/run/containerd/ nertct:v1
second：nerdctl --namespace=k8s.io --insecure-registry=true --runtime=io.containerd.runtime.v1.linux run nertct:v1（in container）
error： failed to mount {Type:overlay Source:overlay Target: Options:[index=off lowerdir=/var/lib/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots/25/fs:/var/lib/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots/24/fs:/var/lib/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots/23/fs:/var/lib/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots/22/fs:/var/lib/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots/21/fs:/var/lib/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots/20/fs:/var/lib/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots/19/fs:/var/lib/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots/18/fs:/var/lib/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots/17/fs:/var/lib/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots/16/fs:/var/lib/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots/15/fs:/var/lib/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots/14/fs:/var/lib/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots/13/fs:/var/lib/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots/12/fs:/var/lib/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots/11/fs:/var/lib/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots/10/fs:/var/lib/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots/9/fs:/var/lib/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots/8/fs:/var/lib/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots/7/fs:/var/lib/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots/6/fs:/var/lib/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots/5/fs:/var/lib/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots/4/fs:/var/lib/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots/3/fs:/var/lib/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots/2/fs]} on "/tmp/initialC3265259058": no such file or directory

[xrow]

Hi have the same issue but with aks agents.

[apostasie]

@xrow need more details to be actionable (version, host info, commands being run). Can you open a ticket with these? Thanks.