[Upgrade to OKD SCOS 4.19] Lot of failures due to "Unsupported OS ID=centos"

[manfuin]

Suddenly OS ID becomes centos instead of scos after update (node reboot by MCO).

This cause a lot of critical system workload failures, like OVN and Multus, maybe there are more. Cluster is falling apart after this upgrade.

$ oc logs ovnkube-node-lh2ph  -n openshift-ovn-kubernetes ovnkube-controller | grep FATAL
+ log cnibincopy 'FATAL ERROR: Unsupported OS ID=centos'
+ echo '2025-06-12T21:37:16+00:00 [{cnibincopy}] FATAL ERROR: Unsupported OS ID=centos'
2025-06-12T21:37:16+00:00 [{cnibincopy}] FATAL ERROR: Unsupported OS ID=centos

$ oc logs -n openshift-multus multus-2bpsl                       
2025-06-12T23:48:19+00:00 [cnibincopy] FATAL ERROR: Unsupported OS ID=centos

OKD SCOS 4.18 /etc/os-release on nodes:

NAME="CentOS Stream CoreOS"
ID="scos"
ID_LIKE="rhel fedora"
VERSION="418.9.202504250632-0"
VERSION_ID="4.18"
VARIANT="CoreOS"
VARIANT_ID=coreos
PLATFORM_ID="platform:el9"
PRETTY_NAME="CentOS Stream CoreOS 418.9.202504250632-0"
ANSI_COLOR="0;31"
CPE_NAME="cpe:/o:centos:centos:9::coreos"
HOME_URL="https://centos.org/"
DOCUMENTATION_URL="https://docs.okd.io/latest/welcome/index.html"
BUG_REPORT_URL="https://access.redhat.com/labs/rhir/"
REDHAT_BUGZILLA_PRODUCT="OpenShift Container Platform"
REDHAT_BUGZILLA_PRODUCT_VERSION="4.18"
REDHAT_SUPPORT_PRODUCT="OpenShift Container Platform"
REDHAT_SUPPORT_PRODUCT_VERSION="4.18"
OPENSHIFT_VERSION="4.18"
OSTREE_VERSION="418.9.202504250632-0"

OKD SCOS 4.19 /etc/os-release on nodes:

NAME="CentOS Stream CoreOS"
VERSION="9.0.20250507-0"
ID="centos"
ID_LIKE="rhel fedora"
VERSION_ID="9"
PLATFORM_ID="platform:el9"
PRETTY_NAME="CentOS Stream CoreOS 9.0.20250507-0"
ANSI_COLOR="0;31"
LOGO="fedora-logo-icon"
CPE_NAME="cpe:/o:centos:centos:9"
HOME_URL="https://centos.org/"
BUG_REPORT_URL="https://issues.redhat.com/"
REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9"
REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream"
OSTREE_VERSION='9.0.20250507-0'
VARIANT=CoreOS
VARIANT_ID=coreos
OPENSHIFT_VERSION="4.19"

[manfuin]

The part of the scripts inside affected containers is:

cni-bin-copy()
{
  # collect host os information
  . /host/etc/os-release
  rhelmajor=
  # detect which version we're using in order to copy the proper binaries
  case "${ID}" in
    rhcos|scos)
      RHEL_VERSION=$(echo "${CPE_NAME}" | cut -f 5 -d :)
      rhelmajor=$(echo $RHEL_VERSION | sed -E 's/([0-9]+)\.{1}[0-9]+(\.[0-9]+)?/\1/')
    ;;
    rhel) rhelmajor=$(echo "${VERSION_ID}" | cut -f 1 -d .)
    ;;
    fedora)
      if [ "${VARIANT_ID}" == "coreos" ]; then
        rhelmajor=8
      else
        log "cnibincopy" "FATAL ERROR: Unsupported Fedora variant=${VARIANT_ID}"
        exit 1
      fi
    ;;
    *) log "cnibincopy" "FATAL ERROR: Unsupported OS ID=${ID}"; exit 1
    ;;
  esac
...

They are clearly not handling the centos ID as it is now.

[nate-duke]

This looks like the source for that function.

https://github.com/openshift/cluster-network-operator/blob/2dc3099a8689a5df9797fe9c14257d7b06886741/bindata/network/ovn-kubernetes/common/008-script-lib.yaml#L438

I'm still looking for where the SCOS builds/repos are since it seems maybe that changing back(?) to centos is the source of the problem.

[manfuin]

The issue is valid for latest 4.19.0-okd-scos.3.

We managed to complete the upgrade process with mitigation - changing the ID to "scos" in /etc/os-release using following one-liner after node got stuck following MCO-initiated reboot:

rm /etc/os-release && cat /usr/lib/os-release | sed '/^ID="centos"/s/.*/ID="scos"/' > /etc/os-release

Afterwards we applied a machineconfig that is provisioning systemd unit, doing the same mitigation on boottime to act on subsequent possible base OS upgrades.

I hope all affected containers will be fixed upstream to support "centos" OS ID, or it will be changed back to "scos" in OKD builds to make this kind of mitigations unnecessary to run OKD 4.19+.

[nate-duke]

It seems like this is certainly important enough a Bug/Issue so that it doesn't get missed or ignored. Does anyone have a clear read on where we can file such an issue to make sure it gets addressed appropriately?

https://github.com/okd-project/okd-scos is archived.
https://github.com/okd-project/images is archived.

Maybe https://github.com/openshift/ovn-kubernetes but will they care about OKD or SCOS?

according to https://github.com/coreos/rhel-coreos-config

Please report OKD or CentOS Stream CoreOS issues in the OKD issue tracker.

So maybe it just belongs on https://github.com/openshift/okd/issues