container install-cni (init) in calico pod Unable to create token for CNI kubeconfig error=Post dial tcp 10.233.0.1:443: i/o timeout

[amir-bialek]

Hey all,

Cluster on-prem, deployed with kubespray.
All is working well except for one worker-node, where the calico-node-2whvs pod is stuck in Init:CrashLoopBackOff .
The logs with ```kubectl -n kube-system logs calico-node-2whvs --container install-cni`` shows:

2024-09-03 14:57:18.963 [FATAL][1] cni-installer/<nil> <nil>: Unable to create token for CNI kubeconfig error=Post "https://10.233.0.1:443/api/v1/namespaces/kube-system/serviceaccounts/calico-cni-plugin/token": dial tcp 10.233.0.1:443: i/o timeout

I will add the the node shows as ready, and there is ping between the master and worker-node.

Exec into the node and run the curl command

 curl --cacert path-to-.crt  \
     --cert path-to.pem \
     --key path-to.key \
   https://10.233.0.1:443/api/v1/namespaces/default/services/kubernetes

give me a timeout, but from other nodes it give a respond.

Can anyone help?

[caseydavenport]

Is kube-proxy running and healthy on that node?

kube-proxy is responsible for programming the dataplane to make 10.233.0.1 functional, so it sounds likely to be an issue with kube-proxy.

This is assuming that:

• You're not using Calico in eBPF mode
• 10.233.0.1 is the ClusterIP of your kubernetes.default Service

[amir-bialek]

Hey @caseydavenport
Thank you for the answer.
Yes, kube-proxy is running. and the ip is the kubernetes. default service.

When I look at the logs of kube-proxy on the broken-node they seems ok:

...
I0905 05:37:55.516524       1 shared_informer.go:311] Waiting for caches to sync for node config
I0905 05:37:55.516524       1 shared_informer.go:311] Waiting for caches to sync for service config
I0905 05:37:55.516524       1 shared_informer.go:311] Waiting for caches to sync for endpoint slice config
I0905 05:37:55.616705       1 shared_informer.go:318] Caches are synced for service config
I0905 05:37:55.616804       1 shared_informer.go:318] Caches are synced for endpoint slice config
I0905 05:37:55.616802       1 shared_informer.go:318] Caches are synced for node config

You wrote that kube-proxy is responsible for programming the dataplane to make 10.233.0.1 functional, can you explain this a bit more?