[HELP] How to configure cluster behind a corporate proxy with custom certificates?

[thony-murray]

I am behind a corporate proxy that uses custom certificates. When I start a cluster, the kube-system pods remain in ContainerCreating forever. I suspect it is due kubernetes not trusting the custom certificates, but I don't know how to fix it.

Logs

~ ❯ k3d version
k3d version v4.3.0
k3s version latest (default)

~ ❯ k3d cluster create MYCLUSTER
INFO[0000] Prep: Network                                
INFO[0000] Created network 'k3d-MYCLUSTER'              
INFO[0000] Created volume 'k3d-MYCLUSTER-images'        
INFO[0001] Creating node 'k3d-MYCLUSTER-server-0'       
INFO[0001] Creating LoadBalancer 'k3d-MYCLUSTER-serverlb' 
INFO[0001] Starting cluster 'MYCLUSTER'                 
INFO[0001] Starting servers...                          
INFO[0001] Starting Node 'k3d-MYCLUSTER-server-0'       
INFO[0008] Starting agents...                           
INFO[0008] Starting helpers...                          
INFO[0008] Starting Node 'k3d-MYCLUSTER-serverlb'       
INFO[0009] (Optional) Trying to get IP of the docker host and inject it into the cluster as 'host.k3d.internal' for easy access 
ERRO[0010] Exec process in node 'k3d-MYCLUSTER-server-0' failed with exit code '1' 
WARN[0010] Failed to get HostIP: Failed to read address for 'host.docker.internal' from nslookup response 
INFO[0010] Cluster 'MYCLUSTER' created successfully!    
INFO[0010] --kubeconfig-update-default=false --> sets --kubeconfig-switch-context=false 
INFO[0010] You can now use it like this:                
kubectl config use-context k3d-MYCLUSTER
kubectl cluster-info

~ ❯ k3d cluster list 
NAME        SERVERS   AGENTS   LOADBALANCER
MYCLUSTER   1/1       0/0      true

~ ❯ kubectl cluster-info
Kubernetes master is running at https://0.0.0.0:59118
CoreDNS is running at https://0.0.0.0:59118/api/v1/namespaces/kube-system/services/kube-dns:dns/proxy
Metrics-server is running at https://0.0.0.0:59118/api/v1/namespaces/kube-system/services/https:metrics-server:/proxy

To further debug and diagnose cluster problems, use 'kubectl cluster-info dump'.

~ ❯ kubectl get pods --all-namespaces
NAMESPACE     NAME                                     READY   STATUS              RESTARTS   AGE
kube-system   metrics-server-7566d596c8-kmsmf          0/1     ContainerCreating   0          62s
kube-system   coredns-7944c66d8d-d9244                 0/1     ContainerCreating   0          62s
kube-system   local-path-provisioner-6d59f47c7-6bdjb   0/1     ContainerCreating   0          62s
kube-system   helm-install-traefik-sfqcd               0/1     ContainerCreating   0          62s

~ ❯ kubectl get events --all-namespaces
NAMESPACE     LAST SEEN   TYPE      REASON                    OBJECT                                        MESSAGE
default       90s         Normal    Starting                  node/k3d-mycluster-server-0                   Starting kubelet.
default       90s         Warning   InvalidDiskCapacity       node/k3d-mycluster-server-0                   invalid capacity 0 on image filesystem
default       90s         Normal    NodeAllocatableEnforced   node/k3d-mycluster-server-0                   Updated Node Allocatable limit across pods
default       90s         Normal    NodeHasSufficientPID      node/k3d-mycluster-server-0                   Node k3d-mycluster-server-0 status is now: NodeHasSufficientPID
default       90s         Normal    NodeHasNoDiskPressure     node/k3d-mycluster-server-0                   Node k3d-mycluster-server-0 status is now: NodeHasNoDiskPressure
default       90s         Normal    NodeHasSufficientMemory   node/k3d-mycluster-server-0                   Node k3d-mycluster-server-0 status is now: NodeHasSufficientMemory
default       89s         Normal    Starting                  node/k3d-mycluster-server-0                   Starting kube-proxy.
default       80s         Normal    NodeReady                 node/k3d-mycluster-server-0                   Node k3d-mycluster-server-0 status is now: NodeReady
kube-system   79s         Normal    ScalingReplicaSet         deployment/local-path-provisioner             Scaled up replica set local-path-provisioner-6d59f47c7 to 1
kube-system   79s         Normal    ScalingReplicaSet         deployment/metrics-server                     Scaled up replica set metrics-server-7566d596c8 to 1
kube-system   79s         Normal    ScalingReplicaSet         deployment/coredns                            Scaled up replica set coredns-7944c66d8d to 1
default       79s         Normal    RegisteredNode            node/k3d-mycluster-server-0                   Node k3d-mycluster-server-0 event: Registered Node k3d-mycluster-server-0 in Controller
kube-system   79s         Normal    SuccessfulCreate          replicaset/coredns-7944c66d8d                 Created pod: coredns-7944c66d8d-d9244
kube-system   79s         Normal    SuccessfulCreate          replicaset/metrics-server-7566d596c8          Created pod: metrics-server-7566d596c8-kmsmf
kube-system   79s         Normal    Scheduled                 pod/metrics-server-7566d596c8-kmsmf           Successfully assigned kube-system/metrics-server-7566d596c8-kmsmf to k3d-mycluster-server-0
kube-system   79s         Normal    SuccessfulCreate          replicaset/local-path-provisioner-6d59f47c7   Created pod: local-path-provisioner-6d59f47c7-6bdjb
kube-system   79s         Normal    Scheduled                 pod/coredns-7944c66d8d-d9244                  Successfully assigned kube-system/coredns-7944c66d8d-d9244 to k3d-mycluster-server-0
kube-system   79s         Normal    Scheduled                 pod/local-path-provisioner-6d59f47c7-6bdjb    Successfully assigned kube-system/local-path-provisioner-6d59f47c7-6bdjb to k3d-mycluster-server-0
kube-system   79s         Normal    SuccessfulCreate          job/helm-install-traefik                      Created pod: helm-install-traefik-sfqcd
kube-system   79s         Normal    Scheduled                 pod/helm-install-traefik-sfqcd                Successfully assigned kube-system/helm-install-traefik-sfqcd to k3d-mycluster-server-0
kube-system   13s         Warning   FailedCreatePodSandBox    pod/metrics-server-7566d596c8-kmsmf           Failed to create pod sandbox: rpc error: code = Unknown desc = failed to get sandbox image "docker.io/rancher/pause:3.1": failed to pull image "docker.io/rancher/pause:3.1": failed to pull and unpack image "docker.io/rancher/pause:3.1": failed to resolve reference "docker.io/rancher/pause:3.1": failed to do request: Head https://registry-1.docker.io/v2/rancher/pause/manifests/3.1: x509: certificate signed by unknown authority
kube-system   9s          Warning   FailedCreatePodSandBox    pod/local-path-provisioner-6d59f47c7-6bdjb    Failed to create pod sandbox: rpc error: code = Unknown desc = failed to get sandbox image "docker.io/rancher/pause:3.1": failed to pull image "docker.io/rancher/pause:3.1": failed to pull and unpack image "docker.io/rancher/pause:3.1": failed to resolve reference "docker.io/rancher/pause:3.1": failed to do request: Head https://registry-1.docker.io/v2/rancher/pause/manifests/3.1: x509: certificate signed by unknown authority
kube-system   0s          Warning   FailedCreatePodSandBox    pod/coredns-7944c66d8d-d9244                  Failed to create pod sandbox: rpc error: code = Unknown desc = failed to get sandbox image "docker.io/rancher/pause:3.1": failed to pull image "docker.io/rancher/pause:3.1": failed to pull and unpack image "docker.io/rancher/pause:3.1": failed to resolve reference "docker.io/rancher/pause:3.1": failed to do request: Head https://registry-1.docker.io/v2/rancher/pause/manifests/3.1: x509: certificate signed by unknown authority
kube-system   0s          Warning   FailedCreatePodSandBox    pod/helm-install-traefik-sfqcd                Failed to create pod sandbox: rpc error: code = Unknown desc = failed to get sandbox image "docker.io/rancher/pause:3.1": failed to pull image "docker.io/rancher/pause:3.1": failed to pull and unpack image "docker.io/rancher/pause:3.1": failed to resolve reference "docker.io/rancher/pause:3.1": failed to do request: Head https://registry-1.docker.io/v2/rancher/pause/manifests/3.1: x509: certificate signed by unknown authority

System

I am on

• macOS Catalina version 10.15.7
• Docker version 20.10.5

[iwilltry42]

Hi @tonyfm15 , thanks for opening this issue!
I converted it to a discussion and cleaned up a bit.

I never used k3d behind a corporate proxy but I know, that several people hit this issue already (see e.g. #184 & #423).

I saw some comments saying, that you can try to mount your certificates into /etc/ssl/certs/, e.g. via k3d cluster create MYCLUSTER --volume /path/to/your/certs.crt:/etc/ssl/certs/yourcert.crt

[uknbr]

Also worked fine for me, thanks a lot!

[Wizmll]

Hey, I wonder if you can help with this (knowing that im working on Debian), I have successfully downloaded the self-signed certificate from our corporate proxy, which is integrated into the firewall. However, during the installation of the K3s master node, I encountered an issue where the Kubernetes pods are stuck on the "ContainerCreating" status and do not progress further. Based on my investigation, I suspect that Kubernetes is not trusting the self-signed certificate from the corporate proxy. This might be causing communication errors between the master node and the pods, resulting in the "ContainerCreating" status.
I kindly request your guidance and expertise to help me resolve this issue.

[iwilltry42]

Hi @MalikaML I saw your comments and your mail.
Did you test the suggestion of the selected answer?
It would be good to have the output of kubectl describe pod for one of those pods and potentially the logs of the K3s container (e.g. docker logs k3d-k3s-default-server-0)

[iwilltry42]

I didn't install docker or k3d on my machine (I don't know if I should ?).

Well, what do you want to do with K3s? Is k3d a suitable solution there?

The SandboxChanged message doesn't really hint on anything proxy-related, but it could be.

How about the logs from the K3s process? You can get them via journalctl -u k3s.
Anyway, we may want to move this to the Rancher Users Slack to not bloat this discussion with stuff that may be unrelated.

[Wizmll]

Hey again, I've read the logs over and over again, only to find that I made a small mistake when editing the calico.yaml, which is that I added "" to true while setting up container_settings (I shouldn't have added the "")
I hope this is clear, thank you so much for your help @iwilltry42