bootkube.service - Worker node does not terminate in bootstrap

[eduardolucioac]

I'm trying to deploy the OKD cluster, but the worker node doesn't finish its installation in the bootstrap process.

Worker node error output:

[   39.613875] ignition[526]: GET https://api-int.mbr.okd.local:22623/config/worker: attempt #11
[   39.621547] ignition[526]: GET result: Internal Server Error

Bootstrap node error output:

[core@localhost ~]$ journalctl -b -f -u release-image.service -u bootkube.service
-- Journal begins at Fri 2021-06-18 21:06:14 UTC. --
Jun 18 23:00:16 localhost bootkube.sh[36767]: Skipped "etcd-ca-bundle-configmap.yaml" configmaps.v1./etcd-ca-bundle -n openshift-config as it already exists
Jun 18 23:00:16 localhost bootkube.sh[36767]: Skipped "etcd-client-secret.yaml" secrets.v1./etcd-client -n openshift-config as it already exists
Jun 18 23:00:43 localhost bootkube.sh[36767]:         Pod Status:openshift-kube-apiserver/kube-apiserver        DoesNotExist
Jun 18 23:00:43 localhost bootkube.sh[36767]:         Pod Status:openshift-kube-scheduler/openshift-kube-scheduler        Ready
Jun 18 23:00:43 localhost bootkube.sh[36767]:         Pod Status:openshift-kube-controller-manager/kube-controller-manager        Ready
Jun 18 23:00:43 localhost bootkube.sh[36767]:         Pod Status:openshift-cluster-version/cluster-version-operator        Failed
Jun 18 23:00:48 localhost bootkube.sh[36767]:         Pod Status:openshift-kube-apiserver/kube-apiserver        DoesNotExist
Jun 18 23:00:48 localhost bootkube.sh[36767]:         Pod Status:openshift-kube-scheduler/openshift-kube-scheduler        Pending
Jun 18 23:00:48 localhost bootkube.sh[36767]:         Pod Status:openshift-kube-controller-manager/kube-controller-manager        Pending
Jun 18 23:00:48 localhost bootkube.sh[36767]:         Pod Status:openshift-cluster-version/cluster-version-operator        Failed

I would appreciate some help here friends!

Thanks! =D

[vrutkovs]

Please attach or link to log bundle

[eduardolucioac]

@vrutkovs

As you asked... =D

log-bundle-20210619133803.tar.gz

Thanks!

[eduardolucioac]

@vrutkovs and anyone who can help... =D

Apart from the log sent (more above), I think I discovered some other informations...

The service on "api" on port "6443" ( https://localhost:6443/api ) is not starting as we can see in these snippets of logs from the service "bootkube.service"...

[core@localhost ~]$ journalctl -b -f -u release-image.service -u bootkube.service
[...]
Jun 19 17:17:03 localhost bootkube.sh[1314]: Starting cluster-bootstrap...
[...]
Jun 19 17:17:47 localhost bootkube.sh[5122]: [#187] failed to fetch discovery: Get "https://localhost:6443/api?timeout=32s": dial tcp [::1]:6443: connect: connection refused
Jun 19 17:17:47 localhost bootkube.sh[5122]: [#188] failed to fetch discovery: Get "https://localhost:6443/api?timeout=32s": dial tcp [::1]:6443: connect: connection refused
Jun 19 17:17:47 localhost bootkube.sh[5122]: [#189] failed to fetch discovery: Get "https://localhost:6443/api?timeout=32s": dial tcp [::1]:6443: connect: connection refused
Jun 19 17:17:47 localhost bootkube.sh[5122]: [#190] failed to fetch discovery: Get "https://localhost:6443/api?timeout=32s": dial tcp [::1]:6443: connect: connection refused
Jun 19 17:17:47 localhost bootkube.sh[5122]: [#191] failed to fetch discovery: Get "https://localhost:6443/api?timeout=32s": dial tcp [::1]:6443: connect: connection refused
Jun 19 17:17:48 localhost bootkube.sh[5122]: [#192] failed to fetch discovery: Get "https://localhost:6443/api?timeout=32s": dial tcp [::1]:6443: connect: connection refused
[...]
Jun 19 17:19:00 localhost bootkube.sh[5122]: Created "99-okd-worker-disable-mitigations.yaml" machineconfigs.v1.machineconfiguration.openshift.io/99-okd-worker-disable-mitigations -n
Jun 19 17:19:00 localhost bootkube.sh[5122]: Failed to create "99_openshift-cluster-api_master-user-data-secret.yaml" secrets.v1./master-user-data -n openshift-machine-api: namespaces "openshift-machine-api" not found
Jun 19 17:19:00 localhost bootkube.sh[5122]: Failed to create "99_openshift-cluster-api_worker-user-data-secret.yaml" secrets.v1./worker-user-data -n openshift-machine-api: namespaces "openshift-machine-api" not found
Jun 19 17:19:00 localhost bootkube.sh[5122]: Created "99_openshift-machineconfig_99-master-ssh.yaml" machineconfigs.v1.machineconfiguration.openshift.io/99-master-ssh -n
Jun 19 17:19:00 localhost bootkube.sh[5122]: Created "99_openshift-machineconfig_99-worker-ssh.yaml" machineconfigs.v1.machineconfiguration.openshift.io/99-worker-ssh -n
Jun 19 17:19:00 localhost bootkube.sh[5122]: [#200] failed to create some manifests:
Jun 19 17:19:00 localhost bootkube.sh[5122]: "99_openshift-cluster-api_master-user-data-secret.yaml": failed to create secrets.v1./master-user-data -n openshift-machine-api: namespaces "openshift-machine-api" not found
Jun 19 17:19:00 localhost bootkube.sh[5122]: "99_openshift-cluster-api_worker-user-data-secret.yaml": failed to create secrets.v1./worker-user-data -n openshift-machine-api: namespaces "openshift-machine-api" not found
Jun 19 17:19:00 localhost bootkube.sh[5122]: Created "99_openshift-cluster-api_master-user-data-secret.yaml" secrets.v1./master-user-data -n openshift-machine-api
Jun 19 17:19:00 localhost bootkube.sh[5122]: Created "99_openshift-cluster-api_worker-user-data-secret.yaml" secrets.v1./worker-user-data -n openshift-machine-api

... is from the excerpt...

Jun 19 17:17:03 localhost bootkube.sh[1314]: Starting cluster-bootstrap...
[...]
Jun 19 17:17:47 localhost bootkube.sh[5122]: [#187] failed to fetch discovery: Get "https://localhost:6443/api?timeout=32s": dial tcp [::1]:6443: connect: connection refused

... that things start to go wrong and finally the final excerpt keeps repeating itself until it reaches the lines...

Jun 19 17:19:00 localhost bootkube.sh[5122]: Created "99_openshift-cluster-api_master-user-data-secret.yaml" secrets.v1./master-user-data -n openshift-machine-api
Jun 19 17:19:00 localhost bootkube.sh[5122]: Created "99_openshift-cluster-api_worker-user-data-secret.yaml" secrets.v1./worker-user-data -n openshift-machine-api

Additionally I tried to access the URLs https://<BOOTSTRAP_LOCAL>:6443 , https://<BOOTSTRAP_IP>:6443 , https://<SERVICES_IP>:6443 and got the result below in all cases...

[core@localhost ~]$ curl -k https://127.0.0.1:6443
{
  "kind": "Status",
  "apiVersion": "v1",
  "metadata": {
    
  },
  "status": "Failure",
  "message": "forbidden: User \"system:anonymous\" cannot get path \"/\"",
  "reason": "Forbidden",
  "details": {
    
  },
  "code": 403
}

. <BOOTSTRAP_LOCAL> - 127.0.0.1 ;
. <BOOTSTRAP_IP> - 10.3.0.4 ;
. <SERVICES_IP> - 10.3.0.14 .

Thanks! =D

[eduardolucioac]

@vrutkovs and anyone who can help... =D

Please attach or link to log bundle

Another "log-bundle", but this time with the three master nodes running...

log-bundle-20210619161810.tar.gz

PLUS: I noticed that the masters nodes initially have the machine name defined as "okd_master_X" ("okd_master_1") , but after a while they restart and this name becomes "localhost". This is normal?

[vrutkovs]

but after a while they restart and this name becomes "localhost". This is normal?

No, this means hostname is not set via DHCP/kernel args. See https://docs.okd.io/latest/installing/installing_bare_metal/installing-bare-metal.html#installation-network-user-infra_installing-bare-metal

[eduardolucioac]

Another "log-bundle", but this time with the three master nodes running..

log-bundle-20210619161810.tar.gz

@vrutkovs

log-bundle-20210619161810.tar.gz

Sorry for insisting, but did you notice anything strange in the logs I sent you? =D

I really don't know what else to do... It's very difficult to solve this problem...

Thanks! =D