stargz-snapshotter doesn't work

[maxpain]

I installed this extension and tried to run a test pod with the estargz image:

apiVersion: v1
kind: Pod
metadata:
  name: nodejs
spec:
  containers:
    - name: nodejs-stargz
      image: ghcr.io/stargz-containers/node:17.8.0-esgz
      command: ["node"]
      args:
        - -e
        - var http = require('http');
          http.createServer(function(req, res) {
          res.writeHead(200);
          res.end('Hello World!\n');
          }).listen(80);
      ports:
        - containerPort: 80

Getting this error:
Error: failed to create containerd task: failed to create shim task: OCI runtime create failed: runc create failed: unable to start container process: exec: "node": executable file not found in $PATH: unknown

I also see these errors in the ext-stargz-snapshotter service.

{"dir":"/var/lib/containerd-stargz-grpc/snapshotter/snapshots/68/fs","error":"specified path \"/var/lib/containerd-stargz-grpc/snapshotter/snapshots/68/fs\" isn't a mountpoint","level":"debug","msg":"failed to unmount","time":"2023-10-06T15:35:52.587369547Z"}

{"error":null,"key":"k8s.io/91/extract-195307888--2O1 sha256:2414385fd51d34e07d564ec6041ee66de902424f028528bce52743d92b1bc875","level":"info","mountpoint":"/var/lib/containerd-stargz-grpc/snapshotter/snapshots/64/fs","msg":"[fusermount fusermount3] not installed; trying direct mount","parent":"sha256:a3926353a4b2389bed133fe4b9f8bdb8439529ba6a965b37ef0c1a7921043a00","time":"2023-10-06T15:34:17.197856631Z"}

[maxpain]

By the way I had the same problem even with fusermount binary and even using nydus-snapshotter

[maxpain]

@rsmitty Have you tested if the stargz-snapshotter extension is working correctly?

[frezbo]

@rsmitty is this relevant anymore?

[ruifung]

It doesn't just not work, it seems to break kubelet ImageGC, and then you'll eventually run out of space.

Warning  ImageGCFailed            2m4s (x2 over 7m4s)    kubelet          get filesystem info: Failed to get the info of the filesystem with mountpoint: failed to get device for dir "/var/lib/containerd/io.containerd.snapshotter.v1.stargz": stat failed on /var/lib/containerd/io.containerd.snapshotter.v1.stargz with error: no such file or directory

[rsmitty]

This is a bug in stargz-snapshotter (or even containerd). See containerd/stargz-snapshotter#1349

[rsmitty]

I was thinking that I may just cheese it and create that path as part of the extension, however. But that will come after I can get some time to figure out why we can't pull from a private registry :/