Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Deploying cluster fails waiting for kubernetes to be ready #440

Open
pixil98 opened this issue Jan 1, 2024 · 1 comment
Open

Deploying cluster fails waiting for kubernetes to be ready #440

pixil98 opened this issue Jan 1, 2024 · 1 comment
Labels
team/area2

Comments

@pixil98
Copy link

pixil98 commented Jan 1, 2024

I've been using this provider to deploy kubernetes clusters on top of Proxmox VMs for a while now. Recently I tried deploying a cluster to do some testing and ran into the following error and now I can't seem to get a cluster to deploy. I've tried on both version 1.4.2 and 1.4.3, same error.

Any ideas on what is going wrong or next steps for troubleshooting?

│ Error:
│ ============= RKE outputs ==============
│ time="2023-12-30T22:16:24Z" level=info msg="[rke_provider] rke cluster changed arguments: map[addons:true ingress:true kubernetes_version:true nodes:true private_registries:true]"
│ time="2023-12-30T22:16:24Z" level=info msg="Creating RKE cluster..."
│ time="2023-12-30T22:16:24Z" level=info msg="Initiating Kubernetes cluster"
│ time="2023-12-30T22:16:24Z" level=info msg="[dialer] Setup tunnel for host [192.168.1.41]"
│ time="2023-12-30T22:16:24Z" level=info msg="[dialer] Setup tunnel for host [192.168.1.40]"
│ time="2023-12-30T22:16:24Z" level=info msg="Finding container [cluster-state-deployer] on host [192.168.1.40], try #1"
│ time="2023-12-30T22:16:24Z" level=info msg="Pulling image [registry.lab.reisman.org/proxy.docker.io/rancher/rke-tools:v0.1.96] on host [192.168.1.40], try #1"
│ time="2023-12-30T22:16:39Z" level=info msg="Image [registry.lab.reisman.org/proxy.docker.io/rancher/rke-tools:v0.1.96] exists on host [192.168.1.40]"
│ time="2023-12-30T22:16:41Z" level=info msg="Starting container [cluster-state-deployer] on host [192.168.1.40], try #1"
│ time="2023-12-30T22:16:41Z" level=info msg="[state] Successfully started [cluster-state-deployer] container on host [192.168.1.40]"
│ time="2023-12-30T22:16:41Z" level=info msg="Finding container [cluster-state-deployer] on host [192.168.1.41], try #1"
│ time="2023-12-30T22:16:41Z" level=info msg="Pulling image [registry.lab.reisman.org/proxy.docker.io/rancher/rke-tools:v0.1.96] on host [192.168.1.41], try #1"
│ time="2023-12-30T22:16:54Z" level=info msg="Image [registry.lab.reisman.org/proxy.docker.io/rancher/rke-tools:v0.1.96] exists on host [192.168.1.41]"
│ time="2023-12-30T22:16:55Z" level=info msg="Starting container [cluster-state-deployer] on host [192.168.1.41], try #1"
│ time="2023-12-30T22:16:56Z" level=info msg="[state] Successfully started [cluster-state-deployer] container on host [192.168.1.41]"
│ time="2023-12-30T22:16:56Z" level=info msg="[certificates] Generating CA kubernetes certificates"
│ time="2023-12-30T22:16:56Z" level=info msg="[certificates] Generating Kubernetes API server aggregation layer requestheader client CA certificates"
│ time="2023-12-30T22:16:57Z" level=info msg="[certificates] GenerateServingCertificate is disabled, checking if there are unused kubelet certificates"
│ time="2023-12-30T22:16:57Z" level=info msg="[certificates] Generating Kubernetes API server certificates"
│ time="2023-12-30T22:16:58Z" level=info msg="[certificates] Generating Service account token key"
│ time="2023-12-30T22:16:58Z" level=info msg="[certificates] Generating Kube Controller certificates"
│ time="2023-12-30T22:16:58Z" level=info msg="[certificates] Generating Kube Scheduler certificates"
│ time="2023-12-30T22:16:58Z" level=info msg="[certificates] Generating Kube Proxy certificates"
│ time="2023-12-30T22:16:59Z" level=info msg="[certificates] Generating Node certificate"
│ time="2023-12-30T22:16:59Z" level=info msg="[certificates] Generating admin certificates and kubeconfig"
│ time="2023-12-30T22:16:59Z" level=info msg="[certificates] Generating Kubernetes API server proxy client certificates"
│ time="2023-12-30T22:17:00Z" level=info msg="[certificates] Generating kube-etcd-192-168-1-40 certificate and key"
│ time="2023-12-30T22:17:01Z" level=info msg="Successfully Deployed state file at [/home/aaron/homelab-deployments/development/terraform-provider-rke-tmp-2841594254/cluster.rkestate]"
│ time="2023-12-30T22:17:01Z" level=info msg="Building Kubernetes cluster"
│ time="2023-12-30T22:17:01Z" level=info msg="[dialer] Setup tunnel for host [192.168.1.40]"
│ time="2023-12-30T22:17:01Z" level=info msg="[dialer] Setup tunnel for host [192.168.1.41]"
│ time="2023-12-30T22:17:01Z" level=info msg="[network] Deploying port listener containers"
│ time="2023-12-30T22:17:01Z" level=info msg="Image [registry.lab.reisman.org/proxy.docker.io/rancher/rke-tools:v0.1.96] exists on host [192.168.1.40]"
│ time="2023-12-30T22:17:02Z" level=info msg="Starting container [rke-etcd-port-listener] on host [192.168.1.40], try #1"
│ time="2023-12-30T22:17:02Z" level=info msg="[network] Successfully started [rke-etcd-port-listener] container on host [192.168.1.40]"
│ time="2023-12-30T22:17:02Z" level=info msg="Image [registry.lab.reisman.org/proxy.docker.io/rancher/rke-tools:v0.1.96] exists on host [192.168.1.40]"
│ time="2023-12-30T22:17:03Z" level=info msg="Starting container [rke-cp-port-listener] on host [192.168.1.40], try #1"
│ time="2023-12-30T22:17:04Z" level=info msg="[network] Successfully started [rke-cp-port-listener] container on host [192.168.1.40]"
│ time="2023-12-30T22:17:04Z" level=info msg="Image [registry.lab.reisman.org/proxy.docker.io/rancher/rke-tools:v0.1.96] exists on host [192.168.1.41]"
│ time="2023-12-30T22:17:04Z" level=info msg="Starting container [rke-worker-port-listener] on host [192.168.1.41], try #1"
│ time="2023-12-30T22:17:05Z" level=info msg="[network] Successfully started [rke-worker-port-listener] container on host [192.168.1.41]"
│ time="2023-12-30T22:17:05Z" level=info msg="[network] Port listener containers deployed successfully"
│ time="2023-12-30T22:17:05Z" level=info msg="[network] Running control plane -> etcd port checks"
│ time="2023-12-30T22:17:05Z" level=info msg="[network] Checking if host [192.168.1.40] can connect to host(s) [192.168.1.40] on port(s) [2379], try #1"
│ time="2023-12-30T22:17:05Z" level=info msg="Image [registry.lab.reisman.org/proxy.docker.io/rancher/rke-tools:v0.1.96] exists on host [192.168.1.40]"
│ time="2023-12-30T22:17:06Z" level=info msg="Starting container [rke-port-checker] on host [192.168.1.40], try #1"
│ time="2023-12-30T22:17:06Z" level=info msg="[network] Successfully started [rke-port-checker] container on host [192.168.1.40]"
│ time="2023-12-30T22:17:06Z" level=info msg="Removing container [rke-port-checker] on host [192.168.1.40], try #1"
│ time="2023-12-30T22:17:06Z" level=info msg="[network] Running control plane -> worker port checks"
│ time="2023-12-30T22:17:06Z" level=info msg="[network] Checking if host [192.168.1.40] can connect to host(s) [192.168.1.41] on port(s) [10250], try #1"
│ time="2023-12-30T22:17:06Z" level=info msg="Image [registry.lab.reisman.org/proxy.docker.io/rancher/rke-tools:v0.1.96] exists on host [192.168.1.40]"
│ time="2023-12-30T22:17:07Z" level=info msg="Starting container [rke-port-checker] on host [192.168.1.40], try #1"
│ time="2023-12-30T22:17:07Z" level=info msg="[network] Successfully started [rke-port-checker] container on host [192.168.1.40]"
│ time="2023-12-30T22:17:07Z" level=info msg="Removing container [rke-port-checker] on host [192.168.1.40], try #1"
│ time="2023-12-30T22:17:07Z" level=info msg="[network] Running workers -> control plane port checks"
│ time="2023-12-30T22:17:07Z" level=info msg="[network] Checking if host [192.168.1.41] can connect to host(s) [192.168.1.40] on port(s) [6443], try #1"
│ time="2023-12-30T22:17:07Z" level=info msg="Image [registry.lab.reisman.org/proxy.docker.io/rancher/rke-tools:v0.1.96] exists on host [192.168.1.41]"
│ time="2023-12-30T22:17:08Z" level=info msg="Starting container [rke-port-checker] on host [192.168.1.41], try #1"
│ time="2023-12-30T22:17:08Z" level=info msg="[network] Successfully started [rke-port-checker] container on host [192.168.1.41]"
│ time="2023-12-30T22:17:08Z" level=info msg="Removing container [rke-port-checker] on host [192.168.1.41], try #1"
│ time="2023-12-30T22:17:08Z" level=info msg="[network] Checking KubeAPI port Control Plane hosts"
│ time="2023-12-30T22:17:08Z" level=info msg="[network] Removing port listener containers"
│ time="2023-12-30T22:17:08Z" level=info msg="Removing container [rke-etcd-port-listener] on host [192.168.1.40], try #1"
│ time="2023-12-30T22:17:08Z" level=info msg="[remove/rke-etcd-port-listener] Successfully removed container on host [192.168.1.40]"
│ time="2023-12-30T22:17:08Z" level=info msg="Removing container [rke-cp-port-listener] on host [192.168.1.40], try #1"
│ time="2023-12-30T22:17:09Z" level=info msg="[remove/rke-cp-port-listener] Successfully removed container on host [192.168.1.40]"
│ time="2023-12-30T22:17:09Z" level=info msg="Removing container [rke-worker-port-listener] on host [192.168.1.41], try #1"
│ time="2023-12-30T22:17:09Z" level=info msg="[remove/rke-worker-port-listener] Successfully removed container on host [192.168.1.41]"
│ time="2023-12-30T22:17:09Z" level=info msg="[network] Port listener containers removed successfully"
│ time="2023-12-30T22:17:09Z" level=info msg="[certificates] Deploying kubernetes certificates to Cluster nodes"
│ time="2023-12-30T22:17:09Z" level=info msg="Finding container [cert-deployer] on host [192.168.1.41], try #1"
│ time="2023-12-30T22:17:09Z" level=info msg="Finding container [cert-deployer] on host [192.168.1.40], try #1"
│ time="2023-12-30T22:17:09Z" level=info msg="Image [registry.lab.reisman.org/proxy.docker.io/rancher/rke-tools:v0.1.96] exists on host [192.168.1.41]"
│ time="2023-12-30T22:17:09Z" level=info msg="Image [registry.lab.reisman.org/proxy.docker.io/rancher/rke-tools:v0.1.96] exists on host [192.168.1.40]"
│ time="2023-12-30T22:17:10Z" level=info msg="Starting container [cert-deployer] on host [192.168.1.41], try #1"
│ time="2023-12-30T22:17:10Z" level=info msg="Starting container [cert-deployer] on host [192.168.1.40], try #1"
│ time="2023-12-30T22:17:10Z" level=info msg="Finding container [cert-deployer] on host [192.168.1.41], try #1"
│ time="2023-12-30T22:17:11Z" level=info msg="Finding container [cert-deployer] on host [192.168.1.40], try #1"
│ time="2023-12-30T22:17:15Z" level=info msg="Finding container [cert-deployer] on host [192.168.1.41], try #1"
│ time="2023-12-30T22:17:15Z" level=info msg="Removing container [cert-deployer] on host [192.168.1.41], try #1"
│ time="2023-12-30T22:17:16Z" level=info msg="Finding container [cert-deployer] on host [192.168.1.40], try #1"
│ time="2023-12-30T22:17:16Z" level=info msg="Removing container [cert-deployer] on host [192.168.1.40], try #1"
│ time="2023-12-30T22:17:16Z" level=info msg="[reconcile] Rebuilding and updating local kube config"
│ time="2023-12-30T22:17:16Z" level=info msg="Successfully Deployed local admin kubeconfig at [/home/aaron/homelab-deployments/development/terraform-provider-rke-tmp-2841594254/kube_config_cluster.yml]"
│ time="2023-12-30T22:17:16Z" level=warning msg="[reconcile] host [192.168.1.40] is a control plane node without reachable Kubernetes API endpoint in the cluster"
│ time="2023-12-30T22:17:16Z" level=warning msg="[reconcile] no control plane node with reachable Kubernetes API endpoint in the cluster found"
│ time="2023-12-30T22:17:16Z" level=info msg="[certificates] Successfully deployed kubernetes certificates to Cluster nodes"
│ time="2023-12-30T22:17:16Z" level=info msg="[file-deploy] Deploying file [/etc/kubernetes/admission.yaml] to node [192.168.1.40]"
│ time="2023-12-30T22:17:16Z" level=info msg="Image [registry.lab.reisman.org/proxy.docker.io/rancher/rke-tools:v0.1.96] exists on host [192.168.1.40]"
│ time="2023-12-30T22:17:16Z" level=info msg="Starting container [file-deployer] on host [192.168.1.40], try #1"
│ time="2023-12-30T22:17:17Z" level=info msg="Successfully started [file-deployer] container on host [192.168.1.40]"
│ time="2023-12-30T22:17:17Z" level=info msg="Waiting for [file-deployer] container to exit on host [192.168.1.40]"
│ time="2023-12-30T22:17:17Z" level=info msg="Waiting for [file-deployer] container to exit on host [192.168.1.40]"
│ time="2023-12-30T22:17:17Z" level=info msg="Container [file-deployer] is still running on host [192.168.1.40]: stderr: [], stdout: []"
│ time="2023-12-30T22:17:18Z" level=info msg="Removing container [file-deployer] on host [192.168.1.40], try #1"
│ time="2023-12-30T22:17:18Z" level=info msg="[remove/file-deployer] Successfully removed container on host [192.168.1.40]"
│ time="2023-12-30T22:17:18Z" level=info msg="[/etc/kubernetes/admission.yaml] Successfully deployed admission control config to Cluster control nodes"
│ time="2023-12-30T22:17:18Z" level=info msg="[file-deploy] Deploying file [/etc/kubernetes/audit-policy.yaml] to node [192.168.1.40]"
│ time="2023-12-30T22:17:18Z" level=info msg="Image [registry.lab.reisman.org/proxy.docker.io/rancher/rke-tools:v0.1.96] exists on host [192.168.1.40]"
│ time="2023-12-30T22:17:19Z" level=info msg="Starting container [file-deployer] on host [192.168.1.40], try #1"
│ time="2023-12-30T22:17:19Z" level=info msg="Successfully started [file-deployer] container on host [192.168.1.40]"
│ time="2023-12-30T22:17:19Z" level=info msg="Waiting for [file-deployer] container to exit on host [192.168.1.40]"
│ time="2023-12-30T22:17:19Z" level=info msg="Waiting for [file-deployer] container to exit on host [192.168.1.40]"
│ time="2023-12-30T22:17:19Z" level=info msg="Container [file-deployer] is still running on host [192.168.1.40]: stderr: [], stdout: []"
│ time="2023-12-30T22:17:20Z" level=info msg="Removing container [file-deployer] on host [192.168.1.40], try #1"
│ time="2023-12-30T22:17:20Z" level=info msg="[remove/file-deployer] Successfully removed container on host [192.168.1.40]"
│ time="2023-12-30T22:17:20Z" level=info msg="[/etc/kubernetes/audit-policy.yaml] Successfully deployed audit policy file to Cluster control nodes"
│ time="2023-12-30T22:17:20Z" level=info msg="[reconcile] Reconciling cluster state"
│ time="2023-12-30T22:17:20Z" level=info msg="[reconcile] This is newly generated cluster"
│ time="2023-12-30T22:17:20Z" level=info msg="Pre-pulling kubernetes images"
│ time="2023-12-30T22:17:20Z" level=info msg="Pulling image [registry.lab.reisman.org/proxy.docker.io/rancher/hyperkube:v1.26.9-rancher1] on host [192.168.1.40], try #1"
│ time="2023-12-30T22:17:20Z" level=info msg="Pulling image [registry.lab.reisman.org/proxy.docker.io/rancher/hyperkube:v1.26.9-rancher1] on host [192.168.1.41], try #1"
│ time="2023-12-30T22:18:26Z" level=info msg="Image [registry.lab.reisman.org/proxy.docker.io/rancher/hyperkube:v1.26.9-rancher1] exists on host [192.168.1.40]"
│ time="2023-12-30T22:18:26Z" level=info msg="Pulling image [registry.lab.reisman.org/proxy.docker.io/rancher/mirrored-pause:3.7] on host [192.168.1.40], try #1"
│ time="2023-12-30T22:18:28Z" level=info msg="Image [registry.lab.reisman.org/proxy.docker.io/rancher/mirrored-pause:3.7] exists on host [192.168.1.40]"
│ time="2023-12-30T22:18:42Z" level=info msg="Image [registry.lab.reisman.org/proxy.docker.io/rancher/hyperkube:v1.26.9-rancher1] exists on host [192.168.1.41]"
│ time="2023-12-30T22:18:42Z" level=info msg="Pulling image [registry.lab.reisman.org/proxy.docker.io/rancher/mirrored-pause:3.7] on host [192.168.1.41], try #1"
│ time="2023-12-30T22:18:43Z" level=info msg="Image [registry.lab.reisman.org/proxy.docker.io/rancher/mirrored-pause:3.7] exists on host [192.168.1.41]"
│ time="2023-12-30T22:18:43Z" level=info msg="Kubernetes images pulled successfully"
│ time="2023-12-30T22:18:43Z" level=info msg="[etcd] Building up etcd plane.."
│ time="2023-12-30T22:18:43Z" level=info msg="Image [registry.lab.reisman.org/proxy.docker.io/rancher/rke-tools:v0.1.96] exists on host [192.168.1.40]"
│ time="2023-12-30T22:18:45Z" level=info msg="Starting container [etcd-fix-perm] on host [192.168.1.40], try #1"
│ time="2023-12-30T22:18:45Z" level=info msg="Successfully started [etcd-fix-perm] container on host [192.168.1.40]"
│ time="2023-12-30T22:18:45Z" level=info msg="Waiting for [etcd-fix-perm] container to exit on host [192.168.1.40]"
│ time="2023-12-30T22:18:45Z" level=info msg="Waiting for [etcd-fix-perm] container to exit on host [192.168.1.40]"
│ time="2023-12-30T22:18:46Z" level=info msg="Removing container [etcd-fix-perm] on host [192.168.1.40], try #1"
│ time="2023-12-30T22:18:46Z" level=info msg="[remove/etcd-fix-perm] Successfully removed container on host [192.168.1.40]"
│ time="2023-12-30T22:18:46Z" level=info msg="Pulling image [registry.lab.reisman.org/proxy.docker.io/rancher/mirrored-coreos-etcd:v3.5.6] on host [192.168.1.40], try #1"
│ time="2023-12-30T22:18:56Z" level=info msg="Image [registry.lab.reisman.org/proxy.docker.io/rancher/mirrored-coreos-etcd:v3.5.6] exists on host [192.168.1.40]"
│ time="2023-12-30T22:18:56Z" level=info msg="Starting container [etcd] on host [192.168.1.40], try #1"
│ time="2023-12-30T22:18:57Z" level=info msg="[etcd] Successfully started [etcd] container on host [192.168.1.40]"
│ time="2023-12-30T22:18:57Z" level=info msg="[etcd] Running rolling snapshot container [etcd-rolling-snapshots] on host [192.168.1.40]"
│ time="2023-12-30T22:18:57Z" level=info msg="Image [registry.lab.reisman.org/proxy.docker.io/rancher/rke-tools:v0.1.96] exists on host [192.168.1.40]"
│ time="2023-12-30T22:18:57Z" level=info msg="Starting container [etcd-rolling-snapshots] on host [192.168.1.40], try #1"
│ time="2023-12-30T22:18:58Z" level=info msg="[etcd] Successfully started [etcd-rolling-snapshots] container on host [192.168.1.40]"
│ time="2023-12-30T22:19:03Z" level=info msg="Image [registry.lab.reisman.org/proxy.docker.io/rancher/rke-tools:v0.1.96] exists on host [192.168.1.40]"
│ time="2023-12-30T22:19:04Z" level=info msg="Starting container [rke-bundle-cert] on host [192.168.1.40], try #1"
│ time="2023-12-30T22:19:04Z" level=info msg="[certificates] Successfully started [rke-bundle-cert] container on host [192.168.1.40]"
│ time="2023-12-30T22:19:04Z" level=info msg="Waiting for [rke-bundle-cert] container to exit on host [192.168.1.40]"
│ time="2023-12-30T22:19:04Z" level=info msg="[certificates] successfully saved certificate bundle [/opt/rke/etcd-snapshots//pki.bundle.tar.gz] on host [192.168.1.40]"
│ time="2023-12-30T22:19:04Z" level=info msg="Removing container [rke-bundle-cert] on host [192.168.1.40], try #1"
│ time="2023-12-30T22:19:04Z" level=info msg="Image [registry.lab.reisman.org/proxy.docker.io/rancher/rke-tools:v0.1.96] exists on host [192.168.1.40]"
│ time="2023-12-30T22:19:05Z" level=info msg="Starting container [rke-log-linker] on host [192.168.1.40], try #1"
│ time="2023-12-30T22:19:06Z" level=info msg="[etcd] Successfully started [rke-log-linker] container on host [192.168.1.40]"
│ time="2023-12-30T22:19:06Z" level=info msg="Removing container [rke-log-linker] on host [192.168.1.40], try #1"
│ time="2023-12-30T22:19:06Z" level=info msg="[remove/rke-log-linker] Successfully removed container on host [192.168.1.40]"
│ time="2023-12-30T22:19:06Z" level=info msg="Image [registry.lab.reisman.org/proxy.docker.io/rancher/rke-tools:v0.1.96] exists on host [192.168.1.40]"
│ time="2023-12-30T22:19:07Z" level=info msg="Starting container [rke-log-linker] on host [192.168.1.40], try #1"
│ time="2023-12-30T22:19:07Z" level=info msg="[etcd] Successfully started [rke-log-linker] container on host [192.168.1.40]"
│ time="2023-12-30T22:19:07Z" level=info msg="Removing container [rke-log-linker] on host [192.168.1.40], try #1"
│ time="2023-12-30T22:19:08Z" level=info msg="[remove/rke-log-linker] Successfully removed container on host [192.168.1.40]"
│ time="2023-12-30T22:19:08Z" level=info msg="[etcd] Successfully started etcd plane.. Checking etcd cluster health"
│ time="2023-12-30T22:19:08Z" level=info msg="[etcd] etcd host [192.168.1.40] reported healthy=true"
│ time="2023-12-30T22:19:08Z" level=info msg="[controlplane] Building up Controller Plane.."
│ time="2023-12-30T22:19:08Z" level=info msg="Finding container [service-sidekick] on host [192.168.1.40], try #1"
│ time="2023-12-30T22:19:08Z" level=info msg="Image [registry.lab.reisman.org/proxy.docker.io/rancher/rke-tools:v0.1.96] exists on host [192.168.1.40]"
│ time="2023-12-30T22:19:08Z" level=info msg="Image [registry.lab.reisman.org/proxy.docker.io/rancher/hyperkube:v1.26.9-rancher1] exists on host [192.168.1.40]"
│ time="2023-12-30T22:19:08Z" level=info msg="Starting container [kube-apiserver] on host [192.168.1.40], try #1"
│ time="2023-12-30T22:19:09Z" level=info msg="[controlplane] Successfully started [kube-apiserver] container on host [192.168.1.40]"
│ time="2023-12-30T22:19:09Z" level=info msg="[healthcheck] Start Healthcheck on service [kube-apiserver] on host [192.168.1.40]"
│ time="2023-12-30T22:19:19Z" level=info msg="[healthcheck] service [kube-apiserver] on host [192.168.1.40] is healthy"
│ time="2023-12-30T22:19:19Z" level=info msg="Image [registry.lab.reisman.org/proxy.docker.io/rancher/rke-tools:v0.1.96] exists on host [192.168.1.40]"
│ time="2023-12-30T22:19:20Z" level=info msg="Starting container [rke-log-linker] on host [192.168.1.40], try #1"
│ time="2023-12-30T22:19:21Z" level=info msg="[controlplane] Successfully started [rke-log-linker] container on host [192.168.1.40]"
│ time="2023-12-30T22:19:21Z" level=info msg="Removing container [rke-log-linker] on host [192.168.1.40], try #1"
│ time="2023-12-30T22:19:21Z" level=info msg="[remove/rke-log-linker] Successfully removed container on host [192.168.1.40]"
│ time="2023-12-30T22:19:21Z" level=info msg="Image [registry.lab.reisman.org/proxy.docker.io/rancher/hyperkube:v1.26.9-rancher1] exists on host [192.168.1.40]"
│ time="2023-12-30T22:19:21Z" level=info msg="Starting container [kube-controller-manager] on host [192.168.1.40], try #1"
│ time="2023-12-30T22:19:22Z" level=info msg="[controlplane] Successfully started [kube-controller-manager] container on host [192.168.1.40]"
│ time="2023-12-30T22:19:22Z" level=info msg="[healthcheck] Start Healthcheck on service [kube-controller-manager] on host [192.168.1.40]"
│ time="2023-12-30T22:19:27Z" level=info msg="[healthcheck] service [kube-controller-manager] on host [192.168.1.40] is healthy"
│ time="2023-12-30T22:19:27Z" level=info msg="Image [registry.lab.reisman.org/proxy.docker.io/rancher/rke-tools:v0.1.96] exists on host [192.168.1.40]"
│ time="2023-12-30T22:19:28Z" level=info msg="Starting container [rke-log-linker] on host [192.168.1.40], try #1"
│ time="2023-12-30T22:19:28Z" level=info msg="[controlplane] Successfully started [rke-log-linker] container on host [192.168.1.40]"
│ time="2023-12-30T22:19:28Z" level=info msg="Removing container [rke-log-linker] on host [192.168.1.40], try #1"
│ time="2023-12-30T22:19:28Z" level=info msg="[remove/rke-log-linker] Successfully removed container on host [192.168.1.40]"
│ time="2023-12-30T22:19:28Z" level=info msg="Image [registry.lab.reisman.org/proxy.docker.io/rancher/hyperkube:v1.26.9-rancher1] exists on host [192.168.1.40]"
│ time="2023-12-30T22:19:28Z" level=info msg="Starting container [kube-scheduler] on host [192.168.1.40], try #1"
│ time="2023-12-30T22:19:28Z" level=info msg="[controlplane] Successfully started [kube-scheduler] container on host [192.168.1.40]"
│ time="2023-12-30T22:19:28Z" level=info msg="[healthcheck] Start Healthcheck on service [kube-scheduler] on host [192.168.1.40]"
│ time="2023-12-30T22:19:34Z" level=info msg="[healthcheck] service [kube-scheduler] on host [192.168.1.40] is healthy"
│ time="2023-12-30T22:19:34Z" level=info msg="Image [registry.lab.reisman.org/proxy.docker.io/rancher/rke-tools:v0.1.96] exists on host [192.168.1.40]"
│ time="2023-12-30T22:19:35Z" level=info msg="Starting container [rke-log-linker] on host [192.168.1.40], try #1"
│ time="2023-12-30T22:19:35Z" level=info msg="[controlplane] Successfully started [rke-log-linker] container on host [192.168.1.40]"
│ time="2023-12-30T22:19:35Z" level=info msg="Removing container [rke-log-linker] on host [192.168.1.40], try #1"
│ time="2023-12-30T22:19:35Z" level=info msg="[remove/rke-log-linker] Successfully removed container on host [192.168.1.40]"
│ time="2023-12-30T22:19:35Z" level=info msg="[controlplane] Successfully started Controller Plane.."
│ time="2023-12-30T22:19:35Z" level=info msg="[authz] Creating rke-job-deployer ServiceAccount"
│ time="2023-12-30T22:19:35Z" level=info msg="[authz] rke-job-deployer ServiceAccount created successfully"
│ time="2023-12-30T22:19:35Z" level=info msg="[authz] Creating system:node ClusterRoleBinding"
│ time="2023-12-30T22:19:35Z" level=info msg="[authz] system:node ClusterRoleBinding created successfully"
│ time="2023-12-30T22:19:35Z" level=info msg="[authz] Creating kube-apiserver proxy ClusterRole and ClusterRoleBinding"
│ time="2023-12-30T22:19:35Z" level=info msg="[authz] kube-apiserver proxy ClusterRole and ClusterRoleBinding created successfully"
│ time="2023-12-30T22:19:35Z" level=info msg="Successfully Deployed state file at [/home/aaron/homelab-deployments/development/terraform-provider-rke-tmp-2841594254/cluster.rkestate]"
│ time="2023-12-30T22:19:35Z" level=info msg="[state] Saving full cluster state to Kubernetes"
│
│ Failed running cluster err:[state] Timeout waiting for kubernetes to be ready
│ ========================================
│
│
│   with module.k8s_cluster.rke_cluster.cluster,
│   on .terraform/modules/k8s_cluster/cluster.tf line 11, in resource "rke_cluster" "cluster":
│   11: resource "rke_cluster" "cluster" {
@pixil98 pixil98 changed the title Deploying cluster fails without clear error Deploying cluster fails waiting for kubernetes to be ready Jan 2, 2024
@pixil98
Copy link
Author

pixil98 commented Jan 3, 2024 •
edited
Loading

I enabled debugging and retried the deploy. The logs still show that the last action is Saving full cluster state to Kubernetes, but now I'm getting the following error message.

│ Error:
│ ============= RKE outputs ==============
│ time="2024-01-03T04:36:01Z" level=debug msg="Loading data.json from local source"
│ time="2024-01-03T04:36:01Z" level=debug msg="data.json SHA256 checksum: 90b07a2e3abc57d43ca70fad7dfd980e889463b68af5027887af44363d457088"
│ time="2024-01-03T04:36:01Z" level=debug msg="Loading data.json from local source"
│ time="2024-01-03T04:36:01Z" level=debug msg="data.json SHA256 checksum: 90b07a2e3abc57d43ca70fad7dfd980e889463b68af5027887af44363d457088"
│ time="2024-01-03T04:36:02Z" level=debug msg="Loading data.json from local source"
│ time="2024-01-03T04:36:02Z" level=debug msg="data.json SHA256 checksum: 90b07a2e3abc57d43ca70fad7dfd980e889463b68af5027887af44363d457088"
│ time="2024-01-03T04:36:02Z" level=debug msg="Loading data.json from local source"
│ time="2024-01-03T04:36:02Z" level=debug msg="data.json SHA256 checksum: 90b07a2e3abc57d43ca70fad7dfd980e889463b68af5027887af44363d457088"
│ time="2024-01-03T04:36:02Z" level=info msg="[rke_provider] rke cluster changed arguments: map[addons:true ingress:true kubernetes_version:true nodes:true private_registries:true]"
│ time="2024-01-03T04:36:02Z" level=debug msg="[rke_provider] nodes values old: [] new: [map[address:192.168.1.40 docker_socket: hostname_override: internal_address:192.168.1.40 labels:map[] node_name: port: role:[controlplane etcd] roles: ssh_agent_auth:false ssh_cert: ssh_cert_path: ssh_key:-----BEGIN OPENSSH PRIVATE KEY-----[REDACTED]-----END OPENSSH PRIVATE KEY-----\n ssh_key_path: taints:[] user:aaron] map[address:192.168.1.41 docker_socket: hostname_override: internal_address:192.168.1.41 labels:map[] node_name: port: role:[worker] roles: ssh_agent_auth:false ssh_cert: ssh_cert_path: ssh_key:-----BEGIN OPENSSH PRIVATE KEY-----[REDACTED]-----END OPENSSH PRIVATE KEY-----\n ssh_key_path: taints:[] user:aaron]]"
│ time="2024-01-03T04:36:02Z" level=debug msg="[rke_provider] private_registries values old: [] new: [map[is_default:true password: url:registry.lab.reisman.org/proxy.docker.io user:]]"

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
team/area2
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@pixil98