Add k3s installer, RuntimeClass, and e2e test for the smolvm Kubernetes runtime

deploy/README.md

@@ -0,0 +1,48 @@
+# Deploying smolvm as a Kubernetes runtime
+
+smolvm is a **containerd shim v2** CRI runtime (`io.containerd.smolvm.v2`): every
+pod sandbox boots as its own microVM (the Kata/Firecracker model), so workloads
+get hardware-level isolation instead of a shared kernel.
+
+## Conformance status
+
+`critest` (cri-tools v1.31.1) on the reference node: **82 Passed / 7 Failed /
+24 Skipped** of 113 — at or above the runc baseline (79 passed) on the same host.
+
+The 7 remaining failures are the boundary of *any* VM-based runtime and fail for
+Kata / Firecracker / gVisor too:
+
+| failing test(s) | why a microVM can't pass it |
+| --- | --- |
+| `portforward` ×2 | containerd's portforward dials `127.0.0.1` in the pod netns; the workload listens inside the VM at the pod IP. Needs a sandbox-controller portforward path (planned). |
+| `HostNetwork`, `HostIpc` | a microVM has its own kernel — it cannot share the host's network / IPC namespace. |
+| mount propagation `rprivate` / `rshared` / `rslave` | bidirectional mount propagation across the VM boundary is not possible with a separate guest kernel. |
+
+These are intentional isolation trade-offs, not defects.
+
+## Quick start (k3s)
+
+```sh
+# 1. install the runtime into k3s (idempotent, version-robust)
+sudo deploy/k3s/install-smolvm-k3s.sh
+
+# 2. prove it end-to-end (deploy a pod as a microVM, check logs + exec)
+sudo deploy/k3s/e2e-test.sh
+```
+
+`install-smolvm-k3s.sh` wires the shim into k3s's embedded containerd by reading
+k3s's *generated* config to find the exact CRI plugin path, exports the shim's
+runtime env to k3s, drops the shim onto containerd's PATH, writes a
+`config.toml.tmpl`, registers the `smolvm` RuntimeClass, and labels nodes.
+
+## Manifests
+
+- [`kubernetes/runtimeclass.yaml`](kubernetes/runtimeclass.yaml) — the `smolvm` RuntimeClass.
+- [`kubernetes/example-pod.yaml`](kubernetes/example-pod.yaml) — a smoke pod (`runtimeClassName: smolvm`).
+
+## Prerequisites
+
+The smolvm runtime payload must be installed on each node under
+`$SMOLVM_DATA_DIR` (default `/var/lib/smolvm`): the musl `agent-rootfs`, the
+`smolvm-vmm` boot helper, `lib/` (libkrun), and the `containerd-shim-smolvm-v2`
+binary on `PATH`. A Linux host with KVM (`/dev/kvm`) is required.

deploy/k3s/e2e-test.sh

@@ -0,0 +1,45 @@
+#!/usr/bin/env bash
+# End-to-end validation: a Pod scheduled onto the smolvm runtime boots as a
+# microVM, runs, streams logs, and accepts exec — through the real k3s/k8s API.
+# Run AFTER install-smolvm-k3s.sh.
+#
+#   sudo ./e2e-test.sh
+#
+# Exits non-zero (and prints why) on any failure, so it can gate CI / a demo.
+set -euo pipefail
+HERE=$(cd "$(dirname "$0")" && pwd)
+K="k3s kubectl"
+POD=smolvm-hello
+
+fail() { echo "E2E FAIL: $*" >&2; $K describe pod "$POD" 2>/dev/null | tail -20 >&2 || true; exit 1; }
+
+echo "==> RuntimeClass 'smolvm' registered?"
+$K get runtimeclass smolvm >/dev/null 2>&1 || fail "RuntimeClass smolvm missing (run install-smolvm-k3s.sh)"
+
+echo "==> deploying the smolvm pod"
+$K delete pod "$POD" --ignore-not-found --wait=true >/dev/null 2>&1 || true
+$K apply -f "$HERE/../kubernetes/example-pod.yaml" >/dev/null
+
+echo "==> waiting for Ready (microVM boot)"
+$K wait --for=condition=Ready "pod/$POD" --timeout=150s || fail "pod never became Ready"
+
+echo "==> the pod really runs on the smolvm runtime handler"
+RC=$($K get "pod/$POD" -o jsonpath='{.spec.runtimeClassName}')
+[ "$RC" = smolvm ] || fail "pod runtimeClassName is '$RC', expected smolvm"
+
+echo "==> logs show the workload ran inside a VM kernel"
+LOGS=$($K logs "$POD")
+echo "$LOGS" | sed 's/^/    /'
+echo "$LOGS" | grep -q SMOLVM_K8S_E2E_OK || fail "expected marker not in logs"
+echo "$LOGS" | grep -qi "kernel:" || fail "no kernel line in logs"
+
+echo "==> exec into the running microVM"
+UID_OUT=$($K exec "$POD" -- id 2>/dev/null) || fail "kubectl exec failed"
+echo "    exec id -> $UID_OUT"
+echo "$UID_OUT" | grep -q "uid=" || fail "exec did not return a valid id"
+
+echo "==> tearing the pod (microVM) down"
+$K delete pod "$POD" --wait=true >/dev/null
+
+echo
+echo "E2E PASS: smolvm pod booted as a microVM, ran, logged, exec'd, and tore down cleanly via k3s."

deploy/k3s/install-smolvm-k3s.sh

@@ -0,0 +1,85 @@
+#!/usr/bin/env bash
+# Install smolvm as a k3s container runtime (RuntimeClass "smolvm").
+#
+# Wires the smolvm containerd shim v2 into k3s's embedded containerd so pods with
+# `runtimeClassName: smolvm` boot as per-workload microVMs. Version-robust: it
+# reads k3s's *generated* containerd config to find the exact CRI plugin path
+# (containerd 1.x and 2.x differ) instead of hardcoding it.
+#
+# Prereqs: k3s installed, and the smolvm runtime payload present under
+# $SMOLVM_DATA_DIR (agent-rootfs, smolvm-vmm boot helper, lib/, and the shim
+# binary on PATH). See docs/kubernetes-runtime.md for building the payload.
+#
+#   sudo ./install-smolvm-k3s.sh
+#   sudo k3s kubectl apply -f ../kubernetes/example-pod.yaml   # e2e smoke test
+set -euo pipefail
+
+SMOLVM_DATA_DIR=${SMOLVM_DATA_DIR:-/var/lib/smolvm}
+SHIM=${SHIM:-/usr/local/bin/containerd-shim-smolvm-v2}
+K3S_CTD_DIR=/var/lib/rancher/k3s/agent/etc/containerd
+HERE=$(cd "$(dirname "$0")" && pwd)
+
+[ "$(id -u)" = 0 ] || { echo "run as root (sudo)"; exit 1; }
+command -v k3s >/dev/null || { echo "k3s not found on PATH"; exit 1; }
+
+echo "==> verifying smolvm runtime payload"
+for f in "$SHIM" "$SMOLVM_DATA_DIR/agent-rootfs" "$SMOLVM_DATA_DIR/smolvm-vmm" "$SMOLVM_DATA_DIR/lib"; do
+  [ -e "$f" ] || { echo "  MISSING: $f — install the smolvm runtime payload first"; exit 1; }
+done
+
+echo "==> exporting the shim's env to k3s (the shim inherits k3s -> containerd env)"
+mkdir -p /etc/systemd/system/k3s.service.d
+cat > /etc/systemd/system/k3s.service.d/smolvm.conf <<EOF
+[Service]
+Environment=SMOLVM_DATA_DIR=$SMOLVM_DATA_DIR
+Environment=SMOLVM_AGENT_ROOTFS=$SMOLVM_DATA_DIR/agent-rootfs
+Environment=SMOLVM_BOOT_BINARY=$SMOLVM_DATA_DIR/smolvm-vmm
+Environment=SMOLVM_LIB_DIR=$SMOLVM_DATA_DIR/lib
+EOF
+systemctl daemon-reload
+
+echo "==> ensuring k3s is up and has generated its base containerd config"
+systemctl enable --now k3s
+for _ in $(seq 1 60); do [ -f "$K3S_CTD_DIR/config.toml" ] && break; sleep 2; done
+CFG="$K3S_CTD_DIR/config.toml"
+[ -f "$CFG" ] || { echo "  k3s did not generate $CFG"; exit 1; }
+
+echo "==> detecting the CRI runtimes plugin path k3s uses"
+# e.g. [plugins.'io.containerd.cri.v1.runtime'.containerd.runtimes.runc]  (ctd 2.x)
+#  or  [plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runc]     (ctd 1.x)
+RUNC_HDR=$(grep -E '\.containerd\.runtimes\.runc\]\s*$' "$CFG" | head -1)
+[ -n "$RUNC_HDR" ] || { echo "  couldn't find the runc runtimes block in $CFG"; exit 1; }
+SMOLVM_HDR=${RUNC_HDR/.runtimes.runc]/.runtimes.smolvm]}
+echo "    runc:   $RUNC_HDR"
+echo "    smolvm: $SMOLVM_HDR"
+
+echo "==> writing k3s containerd template (base + smolvm runtime, no [options])"
+# NB: we deliberately omit the runc-style [...options] sub-table — those options
+# (BinaryName/SystemdCgroup) crash our shim; the smolvm shim takes none.
+cat > "$K3S_CTD_DIR/config.toml.tmpl" <<EOF
+{{ template "base" . }}
+
+$SMOLVM_HDR
+  runtime_type = "io.containerd.smolvm.v2"
+EOF
+
+echo "==> putting the shim on k3s containerd's PATH"
+if [ -d /var/lib/rancher/k3s/data/current/bin ]; then
+  ln -sf "$SHIM" /var/lib/rancher/k3s/data/current/bin/containerd-shim-smolvm-v2
+fi
+
+echo "==> restarting k3s to apply the template"
+systemctl restart k3s
+for _ in $(seq 1 60); do k3s kubectl get --raw='/readyz' >/dev/null 2>&1 && break; sleep 2; done
+
+echo "==> labelling nodes + applying the smolvm RuntimeClass"
+k3s kubectl label node --all smolvm-runtime=true --overwrite
+k3s kubectl apply -f "$HERE/../kubernetes/runtimeclass.yaml"
+
+cat <<EOF
+
+smolvm is installed as a k3s runtime.
+  Smoke test:  sudo k3s kubectl apply -f $HERE/../kubernetes/example-pod.yaml
+               sudo k3s kubectl wait --for=condition=Ready pod/smolvm-hello --timeout=120s
+               sudo k3s kubectl logs smolvm-hello   # expect SMOLVM_K8S_E2E_OK + a VM kernel
+EOF

deploy/kubernetes/example-pod.yaml

@@ -0,0 +1,36 @@
+# Minimal end-to-end smoke pod: runs inside a smolvm microVM via the RuntimeClass.
+#
+#   kubectl apply -f example-pod.yaml
+#   kubectl wait --for=condition=Ready pod/smolvm-hello --timeout=120s
+#   kubectl logs smolvm-hello        # -> SMOLVM_K8S_E2E_OK + `uname -a` from the guest
+#   kubectl exec smolvm-hello -- id  # exec into the microVM
+#
+# The `uname` line proves it's a real VM kernel (not the host); the pod is a
+# throwaway per-workload microVM torn down on delete.
+apiVersion: v1
+kind: Pod
+metadata:
+  name: smolvm-hello
+  labels:
+    app: smolvm-hello
+spec:
+  runtimeClassName: smolvm
+  restartPolicy: Never
+  containers:
+    - name: hello
+      image: busybox:1.36
+      command: ["sh", "-c"]
+      args:
+        - |
+          echo "SMOLVM_K8S_E2E_OK"
+          echo "kernel: $(uname -sr)"
+          echo "whoami: $(id)"
+          # keep the pod alive so `kubectl exec` works during the e2e
+          sleep 3600
+      resources:
+        requests:
+          cpu: "250m"
+          memory: "128Mi"
+        limits:
+          cpu: "1"
+          memory: "512Mi"

deploy/kubernetes/runtimeclass.yaml

@@ -0,0 +1,23 @@
+# RuntimeClass that routes pods to the smolvm containerd shim v2, so each pod
+# sandbox boots as a real per-workload microVM (the Kata/Firecracker model).
+#
+#   kubectl apply -f runtimeclass.yaml
+#   # then set `runtimeClassName: smolvm` on any Pod (see example-pod.yaml)
+#
+# `handler` must match the containerd runtime name (config `...runtimes.smolvm`,
+# runtime_type = io.containerd.smolvm.v2). The overhead reflects the per-pod VM
+# footprint so the scheduler accounts for it.
+apiVersion: node.k8s.io/v1
+kind: RuntimeClass
+metadata:
+  name: smolvm
+handler: smolvm
+overhead:
+  podFixed:
+    cpu: "250m"
+    memory: "160Mi"
+# Restrict to nodes that actually have the smolvm runtime installed (label them
+# `smolvm-runtime=true`). Drop this block on a single-node/all-nodes install.
+scheduling:
+  nodeSelector:
+    smolvm-runtime: "true"