Unable to have k8s Running in Cloudstack 4.20.0

[n4l5u0r]

Trying to deploy a k8s from Cloudstack CKS
based on

mgmt servers : 10.10.0.0/20 / 10.20.0.0/20

• mgmt-1 : 10.X0.0.3
• mgmt-2 : 10.X0.0.4

KVM servers: 10.10.0.0/20, 10.20.0.0/20, 10.40.0.0/20

• kvm-1 : 10.X0.0.6
• kvm-2 : 10.X0.0.7

When deploying the overall k8s everything starts, the instances controle + node, the VR responsible of the isolated network : 10.30.0.0/16

the endpoint is never set to Running, remains on Starting and ends in Error.

the endpoint.urlis set to http://10.10.0.3:8080/client/api

the physical network topology is the following:

• Mgmt network : 10.10.0.0/20 : mgmt + kvm
• Storage network : 10.20.0.0/20 : mgmt + kvm
• Public network : 10.40.0.0/20 : kvm
• Guest network : 10.30.0.0/16

I don't know if the problem is coming from the fact that the Mgmt servers are not accessing the Public network, where the k8s network VR is exposing the k8s services, to check the availability of control service availability.

If I have to expose it what is the best practice on Cloudstack side to have working.

when I request the k8s service from a kvm I have it working up:

# curl -sk https://10.40.0.12:6443/version
{
  "major": "1",
  "minor": "32",
  "gitVersion": "v1.32.5",
  "gitCommit": "9894294ef13a5b32803e3ca2c0d620a088cc84d1",
  "gitTreeState": "clean",
  "buildDate": "2025-05-15T09:05:02Z",
  "goVersion": "go1.23.8",
  "compiler": "gc",
  "platform": "linux/amd64"
}```

[Pearl1594]

@n4l5u0r Can you try to SSH into the control node using:
ssh -i <SSH key / MS SSH key> -p 2222 [email protected]

And then execute the following:

sudo -i
kubectl get nodes
kubectl get po -A

[n4l5u0r]

looks like having issues here :

cloud@test-k8S-control-19a3a5cfc67:~$ sudo -i
root@test-k8S-control-19a3a5cfc67:~# kubectl get nodes
NAME                           STATUS   ROLES           AGE   VERSION
test-k8s-control-19a3a5cfc67   Ready    control-plane   63m   v1.32.5
root@test-k8S-control-19a3a5cfc67:~# kubectl get pods -A
NAMESPACE     NAME                                                   READY   STATUS              RESTARTS       AGE
kube-system   calico-kube-controllers-7bfdc5b57c-9vnph               0/1     ContainerCreating   0              56m
kube-system   calico-node-2q75j                                      0/1     CrashLoopBackOff    11 (25m ago)   56m
kube-system   coredns-668d6bf9bc-5v8n5                               0/1     Pending             0              63m
kube-system   coredns-668d6bf9bc-s8blq                               0/1     Pending             0              63m
kube-system   etcd-test-k8s-control-19a3a5cfc67                      1/1     Running             0              64m
kube-system   kube-apiserver-test-k8s-control-19a3a5cfc67            1/1     Running             0              64m
kube-system   kube-controller-manager-test-k8s-control-19a3a5cfc67   1/1     Running             0              64m
kube-system   kube-proxy-lkbjz                                       1/1     Running             0              63m
kube-system   kube-scheduler-test-k8s-control-19a3a5cfc67            1/1     Running             0              64m

root@test-k8S-control-19a3a5cfc67:~# kubectl get event -A
NAMESPACE     LAST SEEN   TYPE      REASON              OBJECT                                          MESSAGE
kube-system   58m         Warning   FailedScheduling    pod/calico-kube-controllers-7bfdc5b57c-9vnph    0/1 nodes are available: 1 node(s) had untolerated taint {node.kubernetes.io/not-ready: }. preemption: 0/1 nodes are available: 1 Preemption is not helpful for scheduling.
kube-system   58m         Normal    Scheduled           pod/calico-kube-controllers-7bfdc5b57c-9vnph    Successfully assigned kube-system/calico-kube-controllers-7bfdc5b57c-9vnph to test-k8s-control-19a3a5cfc67
kube-system   58m         Normal    SuccessfulCreate    replicaset/calico-kube-controllers-7bfdc5b57c   Created pod: calico-kube-controllers-7bfdc5b57c-9vnph
kube-system   58m         Normal    ScalingReplicaSet   deployment/calico-kube-controllers              Scaled up replica set calico-kube-controllers-7bfdc5b57c from 0 to 1
kube-system   58m         Normal    Scheduled           pod/calico-node-2q75j                           Successfully assigned kube-system/calico-node-2q75j to test-k8s-control-19a3a5cfc67
kube-system   58m         Normal    Pulled              pod/calico-node-2q75j                           Container image "docker.io/calico/cni:v3.30.0" already present on machine
kube-system   58m         Normal    Created             pod/calico-node-2q75j                           Created container: upgrade-ipam
kube-system   58m         Normal    Started             pod/calico-node-2q75j                           Started container upgrade-ipam
kube-system   58m         Normal    Pulled              pod/calico-node-2q75j                           Container image "docker.io/calico/cni:v3.30.0" already present on machine
kube-system   58m         Normal    Created             pod/calico-node-2q75j                           Created container: install-cni
kube-system   58m         Normal    Pulled              pod/calico-node-2q75j                           Container image "docker.io/calico/node:v3.30.0" already present on machine
kube-system   58m         Normal    Started             pod/calico-node-2q75j                           Started container mount-bpffs
kube-system   58m         Normal    Pulled              pod/calico-node-2q75j                           Container image "docker.io/calico/node:v3.30.0" already present on machine
kube-system   58m         Normal    Created             pod/calico-node-2q75j                           Created container: calico-node
kube-system   58m         Normal    SuccessfulCreate    daemonset/calico-node                           Created pod: calico-node-2q75j
kube-system   60m         Warning   FailedScheduling    pod/coredns-668d6bf9bc-5v8n5                    0/1 nodes are available: 1 node(s) had untolerated taint {node.kubernetes.io/not-ready: }. preemption: 0/1 nodes are available: 1 Preemption is not helpful for scheduling.
kube-system   58m         Normal    Scheduled           pod/coredns-668d6bf9bc-5v8n5                    Successfully assigned kube-system/coredns-668d6bf9bc-5v8n5 to test-k8s-control-19a3a5cfc67
kube-system   60m         Warning   FailedScheduling    pod/coredns-668d6bf9bc-s8blq                    0/1 nodes are available: 1 node(s) had untolerated taint {node.kubernetes.io/not-ready: }. preemption: 0/1 nodes are available: 1 Preemption is not helpful for scheduling.
kube-system   58m         Normal    Scheduled           pod/coredns-668d6bf9bc-s8blq                    Successfully assigned kube-system/coredns-668d6bf9bc-s8blq to test-k8s-control-19a3a5cfc67

[Pearl1594]

Seems like the worker node did not come up.
Can you log into the worker node: ssh -i <SSH key / MS SSH key> -p 2223 [email protected]

And check if /var/log/daemon.log or /var/log/cloud-init-output.log or systemctl status deploy-kube-system indicate why the worker node failed to come up..