Issue with creating worker nodes on 4.19.0-okd-scos.12 but cluster was installed with "4.15.0-0.okd-2024-02-23-163410

[jawivana]

Hello! I want to scale this cluster and I have increased the number of workers in the machine set and also created a new machine set(still using the same base image) It is first loading up on FCOS as it was the initial CoreOS (same as template) and then pivoting to SCOS. Then I have to manually poweroff the VM and disable secure boot. This is fine as I had done this manually also when upgrading from 4.15 to 4.16 and 4.17. The problem arises when it boots as SCOS the 2nd time.
`[ 2.686026] systemd[1]: systemd-udevd-kernel.socket: Deactivated successfully.
[ OK ] Closed udev Kernel Socket.
[ 2.686247] systemd[1]: Closed udev Kernel Socket.
[ 2.686693] systemd[1]: Startup finished in 742ms (kernel) + 0 (initrd) + 1.917s (userspace) = 2.660s.
Displaying logs from failed units: ostree-prepare-root.service
Aug 13 18:08:45 systemd[1]: Starting OSTree Prepare OS...
Aug 13 18:08:45 ostree-prepare-root[726]: Resolved OSTree target to: /sysroot/ostree/deploy/fedora-coreos/deploy/6a57b08ddf1536ab85156bc593dd23a8caf9f424b39f1183b69c3b6464644d4.1
Aug 13 18:08:45 ostree-prepare-root[726]: sysroot.readonly configuration value: 1 (fs writable: 1)
Aug 13 18:08:45 ostree-prepare-root[726]: ostree-prepare-root: Failed to mount composefs: failed to mount: No such file or directory
Aug 13 18:08:45 systemd[1]: ostree-prepare-root.service: Main process exited, code=exited, status=1/FAILURE
Aug 13 18:08:45 systemd[1]: ostree-prepare-root.service: Failed with result 'exit-code'.
Aug 13 18:08:45 systemd[1]: Failed to start OSTree Prepare OS.
Aug 13 18:08:45 systemd[1]: ostree-prepare-root.service: Triggering OnFailure= dependencies.
//lib/dracut/hooks/emergency/99-emergency-shell.sh: line 84: echo: write error: Input/output error

Generating "/run/initramfs/rdsosreport.txt"

Entering emergency mode. Exit the shell to continue.
Type "journalctl" to view system logs.
You might want to save "/run/initramfs/rdsosreport.txt" to a USB stick or /boot
after mounting them and attach it to a bug report.

/bin/dracut-emergency: line 22: echo: write error: Input/output error
/bin/dracut-emergency: line 23: echo: write error: Input/output error
/bin/dracut-emergency: line 24: echo: write error: Input/output error
/bin/dracut-emergency: line 25: echo: write error: Input/output error
/bin/dracut-emergency: line 26: echo: write error: Input/output error
/bin/dracut-emergency: line 27: echo: write error: Input/output error
/bin/dracut-emergency: line 28: echo: write error: Input/output error`
Any ideas ?

[Kontr3X]

If I see Input output errors, I typically check the hardware drive for issues. Check if it works, when you reboot the whole node (not only the vm). This can help on stucking IO processes temporary. But make sure you do some earlier tests while it is running and backup if possible.

[jawivana]

Hi ! Im not sure its related to any physical hardware. Other OKD nodes are fine, its when trying to launch the 4th one I get this error.
From my understanding its related to the SCOS node having some config for FCOS as can be seen here : Resolved OSTree target to: /sysroot/ostree/deploy/fedora-coreos/deploy/6a57b08ddf1536ab85156bc593dd23a8caf9f424b39f1183b69c3b6464644d4.1

[jawivana]

My solution was to get a base vm template from another cluster ( this was provisioned with 4.19) and use that to create the worker. It worked fine and didnt fail. It would be very useful if we could have access to coreos templates, I had a look around in the documentation , https://okd.io/docs/project/upgrade-notes/from-4-15/fcos-to-scos-migration/ and the linked issue here #2041 and people asked for access to download scos templates. I however didnt find a link to download any.

[SupremeMortal]

If you're looking for the SCOS replacement base images and installation media, it can be found here: https://cloud.centos.org/centos/scos/9/prod/streams/latest/x86_64/