Pods remain in Unknown status after node reboot

[dariofaccin]

Summary

Given a fully operating K8s (installed via snap) and a juju controller bootstrapped on it, rebooting the node results in failure to start pods:

$ sudo k8s kubectl get pods -A                               
NAMESPACE               NAME                                  READY   STATUS             RESTARTS       AGE
controller-k8s          controller-0                          0/3     Unknown            0              6h37m
controller-k8s          modeloperator-65b5675c96-q6cjc        0/1     Unknown            0              6h37m
kube-system             cilium-2878x                          1/1     Running            1 (3m9s ago)   6h47m
kube-system             cilium-operator-c9487b6f6-cj6hk       1/1     Running            1 (3m9s ago)   6h47m
kube-system             ck-storage-rawfile-csi-controller-0   0/2     Unknown            0              6h47m
kube-system             ck-storage-rawfile-csi-node-4pqkg     0/4     Unknown            0              6h47m
kube-system             coredns-fc9c778db-4m87s               0/1     Unknown            0              6h47m
kube-system             kube-multus-ds-l72q9                  0/1     CrashLoopBackOff   5 (57s ago)    6h38m
kube-system             metrics-server-8694c96fb7-bf5kx       0/1     Unknown            0              6h47m
metallb-system          metallb-controller-7f647445fc-m7tgd   0/1     Unknown            0              6h47m
metallb-system          metallb-speaker-9bkq9                 1/1     Running            1 (3m9s ago)   6h47m
sdcore                  amf-0                                 0/2     Unknown            0              87m
sdcore                  amf-1                                 0/2     Unknown            0              87m
sdcore                  amf-2                                 0/2     Unknown            0              87m
sdcore                  ausf-0                                0/2     Unknown            0              6h17m
sdcore                  gnbsim-0                              0/2     Unknown            0              5h35m
sdcore                  grafana-agent-0                       0/2     Unknown            0              6h16m
sdcore                  modeloperator-675bcd7f67-ddmnh        0/1     Unknown            0              6h18m
sdcore                  mongodb-0                             0/2     Unknown            0              6h17m
sdcore                  nms-0                                 0/2     Unknown            0              6h14m
sdcore                  nrf-0                                 0/2     Unknown            0              6h16m
sdcore                  nssf-0                                0/2     Unknown            0              6h17m
sdcore                  pcf-0                                 0/2     Unknown            0              6h17m
sdcore                  router-0                              0/2     Unknown            0              6h17m
sdcore                  self-signed-certificates-0            0/1     Unknown            0              6h15m
sdcore                  smf-0                                 0/2     Unknown            0              6h17m
sdcore                  traefik-0                             0/2     Unknown            0              6h16m
sdcore                  udm-0                                 0/2     Unknown            0              6h17m
sdcore                  udr-0                                 0/2     Unknown            0              6h14m
sdcore                  upf-0                                 0/3     Unknown            0              6h16m

The error message is the following for all the pods:

sdcore                  30s                       Warning   FailedMount               Pod/udm-0                                                                   MountVolume.MountDevice failed for volume "pvc-0d6ae08c-4bdb-4e6e-88a8-f43b7d119bd9" : kubernetes.io/csi: attacher.MountDevice failed to create newCsiDriverClient: driver name rawfile.csi.openebs.io not found in the list of registered CSI drivers

Also, any juju command involving the controller are not responsive (like juju status).

What Should Happen Instead?

Pods are started.

Reproduction Steps

1. Install k8s snap
2. Bootstrap k8s snap
3. Install juju
4. Bootstrap controller
5. Deploy a juju application
6. Reboot system

System information

inspection-report-20250710_171228.tar.gz

Can you suggest a fix?

No response

Are you interested in contributing with a fix?

No response

[rapour]

Hi @dariofaccin
I was able to reproduce this on Multipass, but it was only a temporary phase while the containers were starting up. After a few seconds, everything returned to normal. Does the issue persist on your setup?

[dariofaccin]

Hi @rapour, thanks for reaching out. Yes, the issue persists until I remove the k8s snap, reinstall it, clean juju config and bootstrap again. I also tried to restart k8sd and the whole snap, but nothing changes.