hi, i got stuck on setting up bootstrap, masters and workers. I am not sure if bootkube.service not starting is the exact issue here, but that is the conclusion i got from troubleshooting.
some details about my installation (bootstrap and masters are the same):
Node OS: FCOS 34.20210626.3.1
Node Specs: 16GB RAM, 4 vCPUs, 120GB disk
following my troubleshoot, tell me if you need more info BR.
what i see on my master:
ignition[533]: GET error: Get "https://........:22623/config/master": EOF
ssh in to bootstrap:
This is the bootstrap node; it will be destroyed when the master is fully up.
The primary services are release-image.service followed by bootkube.service. To watch their status, run e.g.
journalctl -b -f -u release-image.service -u bootkube.service
Fedora CoreOS 34.20210626.3.1
############################################################################
WARNING: This system is using cgroups v1. For increased reliability
it is strongly recommended to migrate this system and your workloads
to use cgroups v2. For instructions on how to adjust kernel arguments
to use cgroups v2, see:
https://docs.fedoraproject.org/en-US/fedora-coreos/kernel-args/
To disable this warning, use:
sudo systemctl disable coreos-check-cgroups.service
############################################################################
Tracker: https://github.com/coreos/fedora-coreos-tracker
Discuss: https://discussion.fedoraproject.org/c/server/coreos/
Last login: Wed Jul 21 13:07:21 2021 from 172.17.57.82
[systemd]
Failed Units: 1
crio-configure.service
Manually gathering logs:
[core@localhost ~]$ journalctl -b -f -u release-image.service -u bootkube.service
-- Journal begins at Wed 2021-07-21 12:40:41 UTC. --
Jul 21 12:55:22 localhost systemd[1]: Starting Download the OpenShift Release Image...
Jul 21 12:55:22 localhost release-image-download.sh[848]: Pulling quay.io/openshift-release-dev/ocp-release@sha256:eafdac268e1f65053de423ba4a028e8de5133ab78e7954d76ed838bcf5f4f666...
Jul 21 12:55:32 localhost podman[850]: 2021-07-21 12:55:32.863448543 +0000 UTC m=+10.077218247 system refresh
Jul 21 12:55:32 localhost podman[850]: 2021-07-21 12:55:32.864988287 +0000 UTC m=+10.078757975 image pull quay.io/openshift-release-dev/ocp-release@sha256:eafdac268e1f65053de423ba4a028e8de5133ab78e7954d76ed838bcf5f4f666
Jul 21 12:55:57 localhost release-image-download.sh[850]: 7df5b82029357ab3601bec214038b683e20603bbbd2e99e447aba5e50e56d607
Jul 21 12:55:57 localhost systemd[1]: Finished Download the OpenShift Release Image.
Jul 21 12:56:00 localhost systemd[1]: Dependency failed for Bootstrap a Kubernetes cluster.
Jul 21 12:56:00 localhost systemd[1]: bootkube.service: Job bootkube.service/start failed with result 'dependency'.
^C
[core@localhost ~]$
[core@localhost ~]$ journalctl -b -f -u bootkube.service
-- Journal begins at Wed 2021-07-21 12:40:41 UTC. --
Jul 21 12:56:00 localhost systemd[1]: Dependency failed for Bootstrap a Kubernetes cluster.
Jul 21 12:56:00 localhost systemd[1]: bootkube.service: Job bootkube.service/start failed with result 'dependency'.
[core@localhost ~]$ for pod in $(sudo podman ps -a -q); do sudo podman logs $pod; done
[core@localhost ~]$
[core@localhost ~]$ tail -f /var/lib/containers/storage/overlay-containers//userdata/ctr.log
tail: cannot open '/var/lib/containers/storage/overlay-containers//userdata/ctr.log' for reading: Permission denied
tail: no files remaining
[core@localhost ~]$
[core@localhost ~]$ journalctl -b -f -u kubelet.service -u crio.service
-- Journal begins at Wed 2021-07-21 12:40:41 UTC. --
Jul 21 13:39:22 localhost podman[27664]: 2021-07-21 13:39:22.154416334 +0000 UTC m=+4.990699734 container init fd11ae355c265dc0178f4f63dca02a1fcb93eb233a628c43129b6b68f541165c (image=quay.io/openshift-release-dev/ocp-release@sha256:eafdac268e1f65053de423ba4a028e8de5133ab78e7954d76ed838bcf5f4f666, name=busy_wilbur, io.openshift.release=4.7.19, io.openshift.release.base-image-digest=sha256:3f403b6b99bf58f4723ff72488ad33789f0e09a66279373cf43af52fd1847bb6)
Jul 21 13:39:22 localhost podman[27664]: 2021-07-21 13:39:22.218625683 +0000 UTC m=+5.054909049 container start fd11ae355c265dc0178f4f63dca02a1fcb93eb233a628c43129b6b68f541165c (image=quay.io/openshift-release-dev/ocp-release@sha256:eafdac268e1f65053de423ba4a028e8de5133ab78e7954d76ed838bcf5f4f666, name=busy_wilbur, io.openshift.release=4.7.19, io.openshift.release.base-image-digest=sha256:3f403b6b99bf58f4723ff72488ad33789f0e09a66279373cf43af52fd1847bb6)
Jul 21 13:39:22 localhost podman[27664]: 2021-07-21 13:39:22.218860484 +0000 UTC m=+5.055143858 container attach fd11ae355c265dc0178f4f63dca02a1fcb93eb233a628c43129b6b68f541165c (image=quay.io/openshift-release-dev/ocp-release@sha256:eafdac268e1f65053de423ba4a028e8de5133ab78e7954d76ed838bcf5f4f666, name=busy_wilbur, io.openshift.release=4.7.19, io.openshift.release.base-image-digest=sha256:3f403b6b99bf58f4723ff72488ad33789f0e09a66279373cf43af52fd1847bb6)
Jul 21 13:39:22 localhost podman[27664]: 2021-07-21 13:39:22.342416739 +0000 UTC m=+5.178700128 container died fd11ae355c265dc0178f4f63dca02a1fcb93eb233a628c43129b6b68f541165c (image=quay.io/openshift-release-dev/ocp-release@sha256:eafdac268e1f65053de423ba4a028e8de5133ab78e7954d76ed838bcf5f4f666, name=busy_wilbur)
Jul 21 13:39:22 localhost podman[27719]: 2021-07-21 13:39:22.867357851 +0000 UTC m=+0.537049331 container remove fd11ae355c265dc0178f4f63dca02a1fcb93eb233a628c43129b6b68f541165c (image=quay.io/openshift-release-dev/ocp-release@sha256:eafdac268e1f65053de423ba4a028e8de5133ab78e7954d76ed838bcf5f4f666, name=busy_wilbur, io.openshift.release=4.7.19, io.openshift.release.base-image-digest=sha256:3f403b6b99bf58f4723ff72488ad33789f0e09a66279373cf43af52fd1847bb6)
Jul 21 13:39:22 localhost systemd[27756]: kubelet.service: Failed to locate executable /usr/bin/hyperkube: No such file or directory
Jul 21 13:39:22 localhost systemd[27756]: kubelet.service: Failed at step EXEC spawning /usr/bin/hyperkube: No such file or directory
Jul 21 13:39:22 localhost systemd[1]: kubelet.service: Main process exited, code=exited, status=203/EXEC
Jul 21 13:39:22 localhost systemd[1]: kubelet.service: Failed with result 'exit-code'.
Jul 21 13:39:22 localhost systemd[1]: Failed to start Kubernetes Kubelet.
Jul 21 13:39:33 localhost systemd[1]: kubelet.service: Scheduled restart job, restart counter is at 188.
Jul 21 13:39:33 localhost systemd[1]: Stopped Kubernetes Kubelet.
Jul 21 13:39:33 localhost systemd[1]: Starting Kubernetes Kubelet...
Jul 21 13:39:38 localhost podman[27798]: 2021-07-21 13:39:38.186074151 +0000 UTC m=+4.825154873 container create 30c9722930c2398f9b600cfa2234cd723ac88dd797f05bb66ebf5fc35fdd1664 (image=quay.io/openshift-release-dev/ocp-release@sha256:eafdac268e1f65053de423ba4a028e8de5133ab78e7954d76ed838bcf5f4f666, name=strange_varahamihira, io.openshift.release=4.7.19, io.openshift.release.base-image-digest=sha256:3f403b6b99bf58f4723ff72488ad33789f0e09a66279373cf43af52fd1847bb6)
Jul 21 13:39:38 localhost podman[27798]: 2021-07-21 13:39:38.422853467 +0000 UTC m=+5.061934184 container init 30c9722930c2398f9b600cfa2234cd723ac88dd797f05bb66ebf5fc35fdd1664 (image=quay.io/openshift-release-dev/ocp-release@sha256:eafdac268e1f65053de423ba4a028e8de5133ab78e7954d76ed838bcf5f4f666, name=strange_varahamihira, io.openshift.release.base-image-digest=sha256:3f403b6b99bf58f4723ff72488ad33789f0e09a66279373cf43af52fd1847bb6, io.openshift.release=4.7.19)
Jul 21 13:39:38 localhost podman[27798]: 2021-07-21 13:39:38.495496758 +0000 UTC m=+5.134577450 container start 30c9722930c2398f9b600cfa2234cd723ac88dd797f05bb66ebf5fc35fdd1664 (image=quay.io/openshift-release-dev/ocp-release@sha256:eafdac268e1f65053de423ba4a028e8de5133ab78e7954d76ed838bcf5f4f666, name=strange_varahamihira, io.openshift.release=4.7.19, io.openshift.release.base-image-digest=sha256:3f403b6b99bf58f4723ff72488ad33789f0e09a66279373cf43af52fd1847bb6)
Jul 21 13:39:38 localhost podman[27798]: 2021-07-21 13:39:38.495812001 +0000 UTC m=+5.134892740 container attach 30c9722930c2398f9b600cfa2234cd723ac88dd797f05bb66ebf5fc35fdd1664 (image=quay.io/openshift-release-dev/ocp-release@sha256:eafdac268e1f65053de423ba4a028e8de5133ab78e7954d76ed838bcf5f4f666, name=strange_varahamihira, io.openshift.release.base-image-digest=sha256:3f403b6b99bf58f4723ff72488ad33789f0e09a66279373cf43af52fd1847bb6, io.openshift.release=4.7.19)
Jul 21 13:39:38 localhost podman[27798]: 2021-07-21 13:39:38.625744766 +0000 UTC m=+5.264825443 container died 30c9722930c2398f9b600cfa2234cd723ac88dd797f05bb66ebf5fc35fdd1664 (image=quay.io/openshift-release-dev/ocp-release@sha256:eafdac268e1f65053de423ba4a028e8de5133ab78e7954d76ed838bcf5f4f666, name=strange_varahamihira)
Jul 21 13:39:39 localhost podman[27798]: 2021-07-21 13:39:39.052415493 +0000 UTC m=+5.691496131 container remove 30c9722930c2398f9b600cfa2234cd723ac88dd797f05bb66ebf5fc35fdd1664 (image=quay.io/openshift-release-dev/ocp-release@sha256:eafdac268e1f65053de423ba4a028e8de5133ab78e7954d76ed838bcf5f4f666, name=strange_varahamihira, io.openshift.release=4.7.19, io.openshift.release.base-image-digest=sha256:3f403b6b99bf58f4723ff72488ad33789f0e09a66279373cf43af52fd1847bb6)
Jul 21 13:39:39 localhost systemd[27885]: kubelet.service: Failed to locate executable /usr/bin/hyperkube: No such file or directory
Jul 21 13:39:39 localhost systemd[27885]: kubelet.service: Failed at step EXEC spawning /usr/bin/hyperkube: No such file or directory
Jul 21 13:39:39 localhost systemd[1]: kubelet.service: Main process exited, code=exited, status=203/EXEC
Jul 21 13:39:39 localhost systemd[1]: kubelet.service: Failed with result 'exit-code'.
Jul 21 13:39:39 localhost systemd[1]: Failed to start Kubernetes Kubelet.
Jul 21 13:39:49 localhost systemd[1]: kubelet.service: Scheduled restart job, restart counter is at 189.
Jul 21 13:39:49 localhost systemd[1]: Stopped Kubernetes Kubelet.
Jul 21 13:39:49 localhost systemd[1]: Starting Kubernetes Kubelet...
^C
[core@localhost ~]$
[core@localhost ~]$ sudo tail -f /var/log/containers/*
tail: cannot open '/var/log/containers/*' for reading: No such file or directory
tail: no files remaining
[core@localhost ~]$
hi, i got stuck on setting up bootstrap, masters and workers. I am not sure if bootkube.service not starting is the exact issue here, but that is the conclusion i got from troubleshooting.
some details about my installation (bootstrap and masters are the same):
Node OS: FCOS 34.20210626.3.1
Node Specs: 16GB RAM, 4 vCPUs, 120GB disk
following my troubleshoot, tell me if you need more info BR.
what i see on my master:
ignition[533]: GET error: Get "https://........:22623/config/master": EOF
ssh in to bootstrap:
This is the bootstrap node; it will be destroyed when the master is fully up.
The primary services are release-image.service followed by bootkube.service. To watch their status, run e.g.
journalctl -b -f -u release-image.service -u bootkube.service
Fedora CoreOS 34.20210626.3.1
############################################################################
WARNING: This system is using cgroups v1. For increased reliability
it is strongly recommended to migrate this system and your workloads
to use cgroups v2. For instructions on how to adjust kernel arguments
to use cgroups v2, see:
https://docs.fedoraproject.org/en-US/fedora-coreos/kernel-args/
To disable this warning, use:
sudo systemctl disable coreos-check-cgroups.service
############################################################################
Tracker: https://github.com/coreos/fedora-coreos-tracker
Discuss: https://discussion.fedoraproject.org/c/server/coreos/
Last login: Wed Jul 21 13:07:21 2021 from 172.17.57.82
[systemd]
Failed Units: 1
crio-configure.service
[core@localhost ~]$ journalctl -b -f -u bootkube.service
-- Journal begins at Wed 2021-07-21 12:40:41 UTC. --
Jul 21 12:56:00 localhost systemd[1]: Dependency failed for Bootstrap a Kubernetes cluster.
Jul 21 12:56:00 localhost systemd[1]: bootkube.service: Job bootkube.service/start failed with result 'dependency'.
[core@localhost ~]$ tail -f /var/lib/containers/storage/overlay-containers//userdata/ctr.log
tail: cannot open '/var/lib/containers/storage/overlay-containers//userdata/ctr.log' for reading: Permission denied
tail: no files remaining
[core@localhost ~]$
[core@localhost ~]$ sudo tail -f /var/log/containers/*
tail: cannot open '/var/log/containers/*' for reading: No such file or directory
tail: no files remaining
[core@localhost ~]$