k3s containerd configuration faulty if shim exists in filesystem outside of kwasm dir

[kate-goldenring]

K3s recently added support for configuring containerd to work with shims. Ironically, this makes it harder to use kwasm.

Here k3s now checks if the shim exists in the filesystem. If it find it, it then assumes it is on the users path, updating the containerd.toml.tmpl config to set runtime_type = "io.containerd.spin.v2". This seems to wire containerd to look for the shim on path even if it tries to qualify where with the options section next:

# /var/lib/rancher/k3s/agent/etc/containerd/config.toml.tmpl
[plugins."io.containerd.grpc.v1.cri".containerd.runtimes."spin"]
  runtime_type = "io.containerd.spin.v2"
[plugins."io.containerd.grpc.v1.cri".containerd.runtimes."spin".options]
  BinaryName = "/opt/kwasm/bin/containerd-shim-spin-v2"
  SystemdCgroup = true

The node installer then sees that the config template was updated and opts not to update it with the path to the binary at /opt/kwasm. The runtime_type should be specified as:

[plugins."io.containerd.grpc.v1.cri".containerd.runtimes.spin]
    runtime_type = "/opt/kwasm/bin/containerd-shim-spin-v2"

When you go to run your app, you get the error:

Failed to create pod sandbox: rpc error: code = Unknown desc = failed to create containerd task: failed to start shim: failed to resolve runtime path: runtime "io.containerd.spin.v2" binary not installed "containerd-shim-spin-v2": file does not exist: unknown

Workaround

1. Option 1: Update the /var/lib/rancher/k3s/agent/etc/containerd/config.toml file to point to the shim under the kwasm directory:

# /var/lib/rancher/k3s/agent/etc/containerd/config.toml
....

[plugins."io.containerd.grpc.v1.cri".containerd.runtimes.spin]
    runtime_type = "/opt/kwasm/bin/containerd-shim-spin-v2"

2. Add the /opt/kwasm directory to path