Failed to upgrade fron 4.19.0-okd-scos.19 to 4.20.0-okd-scos.8

[MainMan1998]

Hello,

I'm trying to upgrade my cluster from 4.19.0-okd-scos.19 to 4.20.0-okd-scos.8.
My cluster has 2 masters and 2 workers on CentOS Stream CoreOS 9.0.20250827-0, all are running on proxmox.
I launched the upgrade, all component have been upgraded on 4.20.0-okd-scos.8 except the machine config who is still in 4.19.0-okd-scos.19.
On machine-config-daemon pods I saw the following error:

W1110 19:54:07.048935 2840026 update.go:2596] Failed to update OS to quay.io/okd/scos-content@sha256:bd9bf79f31c21decab45aa24fc4081f5db2ded99a926779c42ac92c24ed5a325 (will retry): error running rpm-ostree rebase --experimental ostree-unverified-registry:quay.io/okd/scos-content@sha256:bd9bf79f31c21decab45aa24fc4081f5db2ded99a926779c42ac92c24ed5a325: error: Sanity-checking final rootfs: bwrap(/usr/bin/true): Child process exited with code 127
: exit status 1
I1110 19:54:12.055983 2840026 image_manager_helper.go:126] Linking rpm-ostree authfile to /etc/mco/internal-registry-pull-secret.json
I1110 19:54:12.056553 2840026 rpm-ostree.go:201] Executing rebase to quay.io/okd/scos-content@sha256:bd9bf79f31c21decab45aa24fc4081f5db2ded99a926779c42ac92c24ed5a325
I1110 19:54:12.056948 2840026 update.go:2640] Running: rpm-ostree rebase --experimental ostree-unverified-registry:quay.io/okd/scos-content@sha256:bd9bf79f31c21decab45aa24fc4081f5db2ded99a926779c42ac92c24ed5a325
W1110 19:54:12.400421 2840026 update.go:2596] Failed to update OS to quay.io/okd/scos-content@sha256:bd9bf79f31c21decab45aa24fc4081f5db2ded99a926779c42ac92c24ed5a325 (will retry): error running rpm-ostree rebase --experimental ostree-unverified-registry:quay.io/okd/scos-content@sha256:bd9bf79f31c21decab45aa24fc4081f5db2ded99a926779c42ac92c24ed5a325: error: Old and new refs are equal: ostree-unverified-registry:quay.io/okd/scos-content@sha256:bd9bf79f31c21decab45aa24fc4081f5db2ded99a926779c42ac92c24ed5a325
: exit status 1
I1110 19:54:22.401581 2840026 image_manager_helper.go:126] Linking rpm-ostree authfile to /etc/mco/internal-registry-pull-secret.json
I1110 19:54:22.401866 2840026 rpm-ostree.go:201] Executing rebase to quay.io/okd/scos-content@sha256:bd9bf79f31c21decab45aa24fc4081f5db2ded99a926779c42ac92c24ed5a325
I1110 19:54:22.401897 2840026 update.go:2640] Running: rpm-ostree rebase --experimental ostree-unverified-registry:quay.io/okd/scos-content@sha256:bd9bf79f31c21decab45aa24fc4081f5db2ded99a926779c42ac92c24ed5a325
W1110 19:54:22.681935 2840026 update.go:2596] Failed to update OS to quay.io/okd/scos-content@sha256:bd9bf79f31c21decab45aa24fc4081f5db2ded99a926779c42ac92c24ed5a325 (will retry): error running rpm-ostree rebase --experimental ostree-unverified-registry:quay.io/okd/scos-content@sha256:bd9bf79f31c21decab45aa24fc4081f5db2ded99a926779c42ac92c24ed5a325: error: Old and new refs are equal: ostree-unverified-registry:quay.io/okd/scos-content@sha256:bd9bf79f31c21decab45aa24fc4081f5db2ded99a926779c42ac92c24ed5a325
: exit status 1
I1110 19:54:42.683671 2840026 image_manager_helper.go:126] Linking rpm-ostree authfile to /etc/mco/internal-registry-pull-secret.json
I1110 19:54:42.684464 2840026 rpm-ostree.go:201] Executing rebase to quay.io/okd/scos-content@sha256:bd9bf79f31c21decab45aa24fc4081f5db2ded99a926779c42ac92c24ed5a325
I1110 19:54:42.684577 2840026 update.go:2640] Running: rpm-ostree rebase --experimental ostree-unverified-registry:quay.io/okd/scos-content@sha256:bd9bf79f31c21decab45aa24fc4081f5db2ded99a926779c42ac92c24ed5a325
W1110 19:54:42.953451 2840026 update.go:2596] Failed to update OS to quay.io/okd/scos-content@sha256:bd9bf79f31c21decab45aa24fc4081f5db2ded99a926779c42ac92c24ed5a325 (will retry): error running rpm-ostree rebase --experimental ostree-unverified-registry:quay.io/okd/scos-content@sha256:bd9bf79f31c21decab45aa24fc4081f5db2ded99a926779c42ac92c24ed5a325: error: Old and new refs are equal: ostree-unverified-registry:quay.io/okd/scos-content@sha256:bd9bf79f31c21decab45aa24fc4081f5db2ded99a926779c42ac92c24ed5a325
: exit status 1
I1110 19:55:22.957062 2840026 image_manager_helper.go:126] Linking rpm-ostree authfile to /etc/mco/internal-registry-pull-secret.json
I1110 19:55:22.957365 2840026 rpm-ostree.go:201] Executing rebase to quay.io/okd/scos-content@sha256:bd9bf79f31c21decab45aa24fc4081f5db2ded99a926779c42ac92c24ed5a325
I1110 19:55:22.957467 2840026 update.go:2640] Running: rpm-ostree rebase --experimental ostree-unverified-registry:quay.io/okd/scos-content@sha256:bd9bf79f31c21decab45aa24fc4081f5db2ded99a926779c42ac92c24ed5a325
W1110 19:55:23.252158 2840026 update.go:2596] Failed to update OS to quay.io/okd/scos-content@sha256:bd9bf79f31c21decab45aa24fc4081f5db2ded99a926779c42ac92c24ed5a325 (will retry): error running rpm-ostree rebase --experimental ostree-unverified-registry:quay.io/okd/scos-content@sha256:bd9bf79f31c21decab45aa24fc4081f5db2ded99a926779c42ac92c24ed5a325: error: Old and new refs are equal: ostree-unverified-registry:quay.io/okd/scos-content@sha256:bd9bf79f31c21decab45aa24fc4081f5db2ded99a926779c42ac92c24ed5a325
: exit status 1
I1110 19:55:23.253032 2840026 update.go:2808] Rolling back applied changes to OS due to error: Failed to update OS to quay.io/okd/scos-content@sha256:bd9bf79f31c21decab45aa24fc4081f5db2ded99a926779c42ac92c24ed5a325 after retries: timed out waiting for the condition
I1110 19:55:23.253392 2840026 update.go:2640] Running: rpm-ostree cleanup -p
Pruned images: 1 (layers: 51)
Freed: 1.9?GB (pkgcache branches: 0)

I tried to find on internet what should be the cause and I found the following thread: #2275
I tried to cancelled the upgrade, I cleaned the rpm-ostree and force the machine config configuration:

rpm-ostree cleanup -b -p -r -m
rm /etc/machine-config-daemon/currentconfig
touch /run/machine-config-daemon-force

Then, I restarted the upgrade, the cluster said that it was already on okd 4.20.0-okd-scos.8.
Then the cluster restarted automatically, I excepted that I will boot on the new version but it's blocked in the same error on the machine config daemon.

When it have been restarted, it said that they have a mismatch on the /usr/local/bin/configure-ovs.sh on the machine-config-daemon pod
So I created the /run/machine-config-daemon-force file to avoid the mismatch, it resolved the mismatch issue.
Then, it blocked with the version issue: Old and new refs are equal

If I understand the log, it said that it have the old and the new version are the same but I don't know why they don't use the latest version.

In the past I tried to upgrade in 4.20.0-okd-scos.6 but I had the same error.

I attached the oc must-gather do complete the log: must-gather.txt
Does someone have an idea ? Actually, I'm lack of idea…

Many thanks for help :)

[fortinj66]

Look at #2275

4.20 is based on centos streams 10 which requires x86-64-v3 support on the chipset. I’n vSphere that had to be enabled.

[MainMan1998]

Hello
Thanks for help, yes, my server seem to be too old, I used old cpu.
I will plan to upgrade the server hardware and will try a new upgrade with the new server,