[OKD-SCOS v4.16] Upgrade from 4.15.0-0.okd-scos-2024-01-18-223523 to 4.16.0-0.okd-scos-2024-07-11-155249 fails: invalid config

[titou10titou10]

(this is in relation to discussion #1960)

Context

Trying to upgrade a SNO (Single Node) v4.15.0-0.okd-scos-2024-01-18-223523 cluster to v4.16.0-0.okd-scos-2024-07-11-155249

• installed cluster:
  • SNO OKD-SCOS v4.15.0-0.okd-scos-2024-01-18-223523 (an "Accepted" version)
  • on baremetal with UPI
  • using agent based installer (ABI)
  • on vm: 40G RAM, 100G SSD, 10 vCPUS on Proxmox (qemu)
• upgrading to 4.16.0-0.okd-scos-2024-07-11-155249 (not "accepted" as there is no such versions yet)

I also have tried for a multi-node cluster, it fails the same way

I've encountered the exact same problem in many other scenarios and many different versions (upgrade from OKD-FCOS v4.15 to OKD-SCOS v4.16, from OKD-FCOS v4.15 to OKD-SCOS v4.17..)

Summary

It fails with the following error:

Could not update customresourcedefinition "infrastructures.config.openshift.io"
  (47 of 895): the object is invalid, possibly due to local cluster configuration

And in cluster-version-operator logs:

spec.validation.openAPIV3Schema.properties[spec].properties[platformSpec].properties[vsphere].properties[ingressIPs].x-kubernetes-validations[0].rule: 
Invalid value: apiextensions.ValidationRule{Rule:"size(self) == 2 && isIP(self[0]) && isIP(self[1]) ? ip(self[0]).family() != ip(self[1]).family() : true", M
Message:"ingressIPs must contain at most one IPv4 address and at most one IPv6 address", MessageExpression:"", 
Reason:(*apiextensions.FieldValueErrorReason)(nil), FieldPath:""}: compilation failed: ERROR: <input>:1:24: undeclared reference to 'isIP' (in container '')

So it tries to validate somevsphereparameter while this is a baremetal UPI install ????

Details

Base cluster information:

oc get clusterversion
NAME      VERSION                               AVAILABLE   PROGRESSING   SINCE   STATUS
version   4.15.0-0.okd-scos-2024-01-18-223523   True        False         37s     Cluster version is 4.15.0-0.okd-scos-2024-01-18-223523

oc get nodes -o wide
NAME      STATUS   ROLES                         AGE   VERSION           INTERNAL-IP    EXTERNAL-IP   OS-IMAGE                                    KERNEL-VERSION          CONTAINER-RUNTIME
okd-sno   Ready    control-plane,master,worker   32m   v1.28.3+20a5764   192.168.5.71   <none>        CentOS Stream CoreOS 415.9.202401181453-0   5.14.0-408.el9.x86_64   cri-o://1.28.2

oc get co
NAME                                       VERSION                               AVAILABLE   PROGRESSING   DEGRADED   SINCE   MESSAGE
authentication                             4.15.0-0.okd-scos-2024-01-18-223523   True        False         False      11m
baremetal                                  4.15.0-0.okd-scos-2024-01-18-223523   True        False         False      22m
cloud-controller-manager                   4.15.0-0.okd-scos-2024-01-18-223523   True        False         False      22m
cloud-credential                           4.15.0-0.okd-scos-2024-01-18-223523   True        False         False      21m
cluster-autoscaler                         4.15.0-0.okd-scos-2024-01-18-223523   True        False         False      22m
config-operator                            4.15.0-0.okd-scos-2024-01-18-223523   True        False         False      28m
console                                    4.15.0-0.okd-scos-2024-01-18-223523   True        False         False      86s
control-plane-machine-set                  4.15.0-0.okd-scos-2024-01-18-223523   True        False         False      22m
csi-snapshot-controller                    4.15.0-0.okd-scos-2024-01-18-223523   True        False         False      28m
dns                                        4.15.0-0.okd-scos-2024-01-18-223523   True        False         False      26m
etcd                                       4.15.0-0.okd-scos-2024-01-18-223523   True        False         False      22m
image-registry                             4.15.0-0.okd-scos-2024-01-18-223523   True        False         False      14m
ingress                                    4.15.0-0.okd-scos-2024-01-18-223523   True        False         False      27m
insights                                   4.15.0-0.okd-scos-2024-01-18-223523   True        False         False      27m
kube-apiserver                             4.15.0-0.okd-scos-2024-01-18-223523   True        False         False      18m
kube-controller-manager                    4.15.0-0.okd-scos-2024-01-18-223523   True        False         False      19m
kube-scheduler                             4.15.0-0.okd-scos-2024-01-18-223523   True        False         False      22m
kube-storage-version-migrator              4.15.0-0.okd-scos-2024-01-18-223523   True        False         False      28m
machine-api                                4.15.0-0.okd-scos-2024-01-18-223523   True        False         False      27m
machine-approver                           4.15.0-0.okd-scos-2024-01-18-223523   True        False         False      22m
machine-config                             4.15.0-0.okd-scos-2024-01-18-223523   True        False         False      27m
marketplace                                4.15.0-0.okd-scos-2024-01-18-223523   True        False         False      27m
monitoring                                 4.15.0-0.okd-scos-2024-01-18-223523   True        False         False      15m
network                                    4.15.0-0.okd-scos-2024-01-18-223523   True        False         False      29m
node-tuning                                4.15.0-0.okd-scos-2024-01-18-223523   True        False         False      27m
openshift-apiserver                        4.15.0-0.okd-scos-2024-01-18-223523   True        False         False      15m
openshift-controller-manager               4.15.0-0.okd-scos-2024-01-18-223523   True        False         False      14m
openshift-samples                          4.15.0-0.okd-scos-2024-01-18-223523   True        False         False      18m
operator-lifecycle-manager                 4.15.0-0.okd-scos-2024-01-18-223523   True        False         False      27m
operator-lifecycle-manager-catalog         4.15.0-0.okd-scos-2024-01-18-223523   True        False         False      27m
operator-lifecycle-manager-packageserver   4.15.0-0.okd-scos-2024-01-18-223523   True        False         False      19m
service-ca                                 4.15.0-0.okd-scos-2024-01-18-223523   True        False         False      28m
storage                                    4.15.0-0.okd-scos-2024-01-18-223523   True        False         False      28m

Force upgrade to v4.16.0-0.okd-scos-2024-07-11-155249

oc adm upgrade --force --allow-explicit-upgrade --to-image=registry.ci.openshift.org/origin/release-scos:4.16.0-0.okd-scos-2024-07-11-155249

warning: Using by-tag pull specs is dangerous, and while we still allow it in combination with --force for backward compatibility, it would be much safer to pass a by-digest pull spec instead
warning: The requested upgrade image is not one of the available updates. You have used --allow-explicit-upgrade for the update to proceed anyway
warning: --force overrides cluster verification of your supplied release image and waives any update precondition failures.
Requested update to release image registry.ci.openshift.org/origin/release-scos:4.16.0-0.okd-scos-2024-07-11-155249

After a few minutes, the upgrade fails:

oc get clusterversion
NAME      VERSION                               AVAILABLE   PROGRESSING   SINCE   STATUS
version   4.15.0-0.okd-scos-2024-01-18-223523   True        True          2m7s    Unable to apply 4.16.0-0.okd-scos-2024-07-11-155249: some cluster configuration is invalid


oc get clusterversion -o yaml
apiVersion: v1
items:
- apiVersion: config.openshift.io/v1
  kind: ClusterVersion
  metadata:
    creationTimestamp: "2024-07-12T14:00:23Z"
    generation: 3
    name: version
    resourceVersion: "27690"
    uid: 5710e759-147c-48bf-abfc-fe4579f56ee7
  spec:
    channel: stable-scos-4
    clusterID: 1e066a1d-66a1-4115-b7d4-aa80d71af058
    desiredUpdate:
      architecture: ""
      force: true
      image: registry.ci.openshift.org/origin/release-scos:4.16.0-0.okd-scos-2024-07-11-155249
      version: ""
    upstream: https://amd64.origin.releases.ci.openshift.org/graph
  status:
    availableUpdates: null
    capabilities:
      enabledCapabilities:
      - Build
      - CSISnapshot
      - CloudControllerManager
      - CloudCredential
      - Console
      - DeploymentConfig
      - ImageRegistry
      - Ingress
      - Insights
      - MachineAPI
      - NodeTuning
      - OperatorLifecycleManager
      - Storage
      - baremetal
      - marketplace
      - openshift-samples
      knownCapabilities:
      - Build
      - CSISnapshot
      - CloudControllerManager
      - CloudCredential
      - Console
      - DeploymentConfig
      - ImageRegistry
      - Ingress
      - Insights
      - MachineAPI
      - NodeTuning
      - OperatorLifecycleManager
      - Storage
      - baremetal
      - marketplace
      - openshift-samples
    conditions:
    - lastTransitionTime: "2024-07-12T14:00:31Z"
      status: "True"
      type: RetrievedUpdates
    - lastTransitionTime: "2024-07-12T14:45:52Z"
      message: Capabilities match configured spec
      reason: AsExpected
      status: "False"
      type: ImplicitlyEnabledCapabilities
    - lastTransitionTime: "2024-07-12T14:00:31Z"
      message: Payload loaded version="4.16.0-0.okd-scos-2024-07-11-155249" image="registry.ci.openshift.org/origin/release-scos:4.16.0-0.okd-scos-2024-07-11-155249"
        architecture="amd64"
      reason: PayloadLoaded
      status: "True"
      type: ReleaseAccepted
    - lastTransitionTime: "2024-07-12T14:40:30Z"
      message: Done applying 4.15.0-0.okd-scos-2024-01-18-223523
      status: "True"
      type: Available
    - lastTransitionTime: "2024-07-12T14:47:22Z"
      message: 'Could not update customresourcedefinition "infrastructures.config.openshift.io"
        (47 of 895): the object is invalid, possibly due to local cluster configuration'
      reason: UpdatePayloadResourceInvalid
      status: "True"
      type: Failing
    - lastTransitionTime: "2024-07-12T14:45:38Z"
      message: 'Unable to apply 4.16.0-0.okd-scos-2024-07-11-155249: some cluster
        configuration is invalid'
      reason: UpdatePayloadResourceInvalid
      status: "True"
      type: Progressing
    desired:
      image: registry.ci.openshift.org/origin/release-scos:4.16.0-0.okd-scos-2024-07-11-155249
      version: 4.16.0-0.okd-scos-2024-07-11-155249
    history:
    - acceptedRisks: |-
        Target release version="" image="registry.ci.openshift.org/origin/release-scos:4.16.0-0.okd-scos-2024-07-11-155249" cannot be verified, but continuing anyway because the update was forced: release images that are not accessed via digest cannot be verified
        Precondition "ClusterVersionRecommendedUpdate" failed because of "UnknownUpdate": RetrievedUpdates=True (), so the update from 4.15.0-0.okd-scos-2024-01-18-223523 to 4.16.0-0.okd-scos-2024-07-11-155249 is probably neither recommended nor supported.
      completionTime: null
      image: registry.ci.openshift.org/origin/release-scos:4.16.0-0.okd-scos-2024-07-11-155249
      startedTime: "2024-07-12T14:45:38Z"
      state: Partial
      verified: false
      version: 4.16.0-0.okd-scos-2024-07-11-155249
    - completionTime: "2024-07-12T14:40:30Z"
      image: quay.io/okd/scos-release@sha256:ca26e01206c31fa9fb74f611cb5258aea48c03b7ce146251d86b984c1ee72942
      startedTime: "2024-07-12T14:00:31Z"
      state: Completed
      verified: false
      version: 4.15.0-0.okd-scos-2024-01-18-223523
    observedGeneration: 3
    versionHash: iZdIgxduII8=
 kind: List
 metadata:
   resourceVersion: ""

infrastructures.config.openshift.io:

oc get infrastructures.config.openshift.io cluster -o yaml

apiVersion: config.openshift.io/v1
kind: Infrastructure
metadata:
  creationTimestamp: "2024-07-12T14:00:17Z"
  generation: 1
  name: cluster
  resourceVersion: "531"
  uid: baae0b43-feff-4c31-a94d-120c595435cb
spec:
  cloudConfig:
    name: ""
  platformSpec:
    type: None
status:
  apiServerInternalURI: https://api-int.okd-sno.denis.prive:6443
  apiServerURL: https://api.okd-sno.denis.prive:6443
  controlPlaneTopology: SingleReplica
  cpuPartitioning: None
  etcdDiscoveryDomain: ""
  infrastructureName: okd-sno-pbbjs
  infrastructureTopology: SingleReplica
  platform: None
  platformStatus:
    type: None

[UPDATED]
After some investigation, theopenshift-cluster-version/cluster-version-operatorpod logs reports related errors

oc logs cluster-version-operator-674f757d7d-gj2hs -n openshift-cluster-version
{...}
  I0712 14:45:54.939774       1 sync_worker.go:1014] Done syncing for customresourcedefinition "images.config.openshift.io" (45 of 895)
  I0712 14:45:54.939956       1 sync_worker.go:999] Running sync for customresourcedefinition "imagetagmirrorsets.config.openshift.io" (46 of 895)
  I0712 14:45:54.998132       1 sync_worker.go:1014] Done syncing for customresourcedefinition "imagetagmirrorsets.config.openshift.io" (46 of 895)
  I0712 14:45:54.998268       1 sync_worker.go:999] Running sync for customresourcedefinition "infrastructures.config.openshift.io" (47 of 895)
  E0712 14:45:55.517832       1 task.go:122] error running apply for customresourcedefinition "infrastructures.config.openshift.io" (47 of 895): CustomResourceDefinition.apiextension
  s.k8s.io "infrastructures.config.openshift.io" is invalid: [spec.validation.openAPIV3Schema.properties[spec].properties[platformSpec].properties[vsphere].properties[apiServerIntern
  alIPs].items.x-kubernetes-validations[0].rule: Invalid value: apiextensions.ValidationRule{Rule:"isIP(self)", Message:"value must be a valid IP address", MessageExpression:"", Reas
  on:(*apiextensions.FieldValueErrorReason)(nil), FieldPath:""}: compilation failed: ERROR: <input>:1:5: undeclared reference to 'isIP' (in container '')
   | isIP(self)
   | ....^, spec.validation.openAPIV3Schema.properties[spec].properties[platformSpec].properties[vsphere].properties[apiServerInternalIPs].x-kubernetes-validations[0].rule: Invalid v
  alue: apiextensions.ValidationRule{Rule:"size(self) == 2 && isIP(self[0]) && isIP(self[1]) ? ip(self[0]).family() != ip(self[1]).family() : true", Message:"apiServerInternalIPs mus
  t contain at most one IPv4 address and at most one IPv6 address", MessageExpression:"", Reason:(*apiextensions.FieldValueErrorReason)(nil), FieldPath:""}: compilation failed: ERROR
  : <input>:1:24: undeclared reference to 'isIP' (in container '')
   | size(self) == 2 && isIP(self[0]) && isIP(self[1]) ? ip(self[0]).family() != ip(self[1]).family() : true
   | .......................^
  ERROR: <input>:1:41: undeclared reference to 'isIP' (in container '')
   | size(self) == 2 && isIP(self[0]) && isIP(self[1]) ? ip(self[0]).family() != ip(self[1]).family() : true
   | ........................................^
  ERROR: <input>:1:55: undeclared reference to 'ip' (in container '')
   | size(self) == 2 && isIP(self[0]) && isIP(self[1]) ? ip(self[0]).family() != ip(self[1]).family() : true
   | ......................................................^
  ERROR: <input>:1:71: undeclared reference to 'family' (in container '')
   | size(self) == 2 && isIP(self[0]) && isIP(self[1]) ? ip(self[0]).family() != ip(self[1]).family() : true
   | ......................................................................^
  ERROR: <input>:1:79: undeclared reference to 'ip' (in container '')
   | size(self) == 2 && isIP(self[0]) && isIP(self[1]) ? ip(self[0]).family() != ip(self[1]).family() : true
   | ..............................................................................^
  ERROR: <input>:1:95: undeclared reference to 'family' (in container '')
   | size(self) == 2 && isIP(self[0]) && isIP(self[1]) ? ip(self[0]).family() != ip(self[1]).family() : true
   | ..............................................................................................^, spec.validation.openAPIV3Schema.properties[spec].properties[platformSpec].proper
  ties[vsphere].properties[ingressIPs].items.x-kubernetes-validations[0].rule: Invalid value: apiextensions.ValidationRule{Rule:"isIP(self)", Message:"value must be a valid IP addres
  s", MessageExpression:"", Reason:(*apiextensions.FieldValueErrorReason)(nil), FieldPath:""}: compilation failed: ERROR: <input>:1:5: undeclared reference to 'isIP' (in container ''
  )
   | isIP(self)
   | ....^, spec.validation.openAPIV3Schema.properties[spec].properties[platformSpec].properties[vsphere].properties[ingressIPs].x-kubernetes-validations[0].rule: Invalid value: apie
  xtensions.ValidationRule{Rule:"size(self) == 2 && isIP(self[0]) && isIP(self[1]) ? ip(self[0]).family() != ip(self[1]).family() : true", Message:"ingressIPs must contain at most on
  e IPv4 address and at most one IPv6 address", MessageExpression:"", Reason:(*apiextensions.FieldValueErrorReason)(nil), FieldPath:""}: compilation failed: ERROR: <input>:1:24: unde
  clared reference to 'isIP' (in container '')

So it seems it is trying to validate somevsphereparameter while this is a baremetal UPI install ????

spec.validation.openAPIV3Schema.properties[spec].properties[platformSpec].properties[vsphere].properties[ingressIPs].x-kubernetes-validations[0].rule: 
Invalid value: apiextensions.ValidationRule{Rule:"size(self) == 2 && isIP(self[0]) && isIP(self[1]) ? ip(self[0]).family() != ip(self[1]).family() : true", M
Message:"ingressIPs must contain at most one IPv4 address and at most one IPv6 address", MessageExpression:"", 
Reason:(*apiextensions.FieldValueErrorReason)(nil), FieldPath:""}: compilation failed: ERROR: <input>:1:24: undeclared reference to 'isIP' (in container '')

[madpearl]

There is acceptable version of 4.16.0-0.okd-scos-2024-07-21-033952 , you can check it, but I thing that you will get the same error, because the difference between 4.15.0-0.okd-scos-2024-01-18-223523 are too many!

[aleskandro]

Hi all,
this error happens because some new validation filters have been set in the CRDs that the CVO applies before updating to the kube-apiserver version supporting them.

Can you help test this procedure on your platforms?

I tested the same path of this thread's title (4.15.0-0.okd-scos-2024-01-18-223523 -> 4.16.0-0.okd-scos-2024-07-11-155249) in a fresh 4.15 OKD on AWS, and it went fine.

1. oc adm upgrade --to-image registry.ci.openshift.org/origin/release-scos:4.16.0-0.okd-scos-2024-07-21-033952 --force --allow-explicit-upgrades
2. Wait for the error to come up:

ERROR: <input>:1:5: undeclared reference to 'isIP'

4. oc scale --replicas=0 deployments/cluster-version-operator -n openshift-cluster-version. The upgrade will stop here.
5. oc adm release info --image-for=hyperkube registry.ci.openshift.org/origin/release-scos:4.16.0-0.okd-scos-2024-07-21-033952
6. oc adm release info --image-for=cluster-kube-apiserver-operator registry.ci.openshift.org/origin/release-scos:4.16.0-0.okd-scos-2024-07-21-033952
7. Force the kube-apiserver upgrade by patching the kube-apiserver-operator. oc edit -n openshift-kube-apiserver-operator deployments/kube-apiserver-operator and change:

• the image of the kube-apiserver-operator container to the one given by point (6)
• the value of the OPERATOR_IMAGE env var of the kube-apiserver-operator container to the one given by point (6)
• the value of the IMAGE env var of the kube-apiserver-operator container to the one given by point (5) [hyperkube]
• the value of the OPERAND_IMAGE_VERSION env var to 1.29.6
• keep the OPERATOR_IMAGE_VERSION as is (4.15.xxx...)

5. Wait for the new kube-apiserver-operator and kube-apiserver to be deployed (all the replicas if non-SNO) by watching at oc get pods -n openshift-kube-apiserver-operator && oc get pods -n openshift-kube-apiserver
6. Once all the replicas of the new kube-apiserver are ready, scale out the CVO: oc scale --replicas=1 deployments/cluster-version-operator -n openshift-cluster-version
7. the upgrade should complete. You can follow with:

watch oc get co && oc get nodes && oc get mcp && oc get clusterversion

[jorgesantanaam]

@JOECHONG0420 in what way can i overtake this machine-config-daemon and error reading osImageURL from rpm-ostree issue with any major problems to the cluster and the nodes ?

[JOECHONG0420]

run "rpm-ostree status" in the fail node and another node

[jorgesantanaam]

Node that failed:

[core@master-1 ~]$ rpm-ostree status
error: Loading sysroot: Failed to invoke RegisterClient: GDBus.Error:org.freedesktop.DBus.Error.NameHasNoOwner: Could not activate remote peer.
[core@master-1 ~]$ sudo rpm-ostree status
A dependency job for rpm-ostreed.service failed. See 'journalctl -xe' for details.
○ rpm-ostreed.service - rpm-ostree System Management Daemon
Loaded: loaded (/usr/lib/systemd/system/rpm-ostreed.service; static)
Drop-In: /run/systemd/system/rpm-ostreed.service.d
└─bug2111817.conf
/etc/systemd/system/rpm-ostreed.service.d
└─mco-controlplane-nice.conf
Active: inactive (dead)
Docs: man:rpm-ostree(1)

Sep 12 01:55:11 master-1 systemd[1]: Dependency failed for rpm-ostree System Management Daemon.
Sep 12 01:55:11 master-1 systemd[1]: rpm-ostreed.service: Job rpm-ostreed.service/start failed with result 'dependency'.
Sep 12 02:00:14 master-1 systemd[1]: Dependency failed for rpm-ostree System Management Daemon.
Sep 12 02:00:14 master-1 systemd[1]: rpm-ostreed.service: Job rpm-ostreed.service/start failed with result 'dependency'.
Sep 12 02:05:19 master-1 systemd[1]: Dependency failed for rpm-ostree System Management Daemon.
Sep 12 02:05:19 master-1 systemd[1]: rpm-ostreed.service: Job rpm-ostreed.service/start failed with result 'dependency'.
Sep 12 02:09:37 master-1 systemd[1]: Dependency failed for rpm-ostree System Management Daemon.
Sep 12 02:09:37 master-1 systemd[1]: rpm-ostreed.service: Job rpm-ostreed.service/start failed with result 'dependency'.
Sep 12 02:09:43 master-1 systemd[1]: Dependency failed for rpm-ostree System Management Daemon.
Sep 12 02:09:43 master-1 systemd[1]: rpm-ostreed.service: Job rpm-ostreed.service/start failed with result 'dependency'.
error: Loading sysroot: exit status: 1

The another node :

[core@master-2 ~]$ rpm-ostree status
State: idle
Deployments:
● ostree-unverified-registry:quay.io/openshift/okd-content@sha256:eb85d903c52970e2d6823d92c880b20609d8e8e0dbc5ad27e16681ff444c8c83
Digest: sha256:b419768333f54726a8bc72929374a0f08b31c1b485872c596b044fb20544339f
Version: 39.20240210.3.0 (2024-03-08T10:08:15Z)

[jorgesantanaam]

Hello @JOECHONG0420 @jbparfum @aleskandro or anyone Else that could help me with this issue?

Followed the guide above, tried to upgrade from 4.15.0-0.okd-2024-03-10-010116 to 4.16.0-0.okd-scos-2024-09-27-110344
Machine config daemon crash looping with
I0930 12:57:49.876855 9599 start.go:68] Version: machine-config-daemon-4.6.0-202006240615.p0-2842-g6a425abb-dirty (6a425abbe5e330f0e7bc0849f4927d4c55d0c9e5) E0930 12:57:49.876997 9599 rpm-ostree.go:276] Merged secret file does not exist; defaulting to cluster pull secret I0930 12:57:49.877022 9599 rpm-ostree.go:263] Linking ostree authfile to /var/lib/kubelet/config.json F0930 12:57:49.942414 9599 start.go:106] Failed to initialize single run daemon: error reading osImageURL from rpm-ostree: exit status 1

[JOECHONG0420]

can you show me the result with command below.
cat /boot/loader/entries/ostree-1.conf
ls /boot -alh

[titou10titou10]

oc get clusterversion
NAME      VERSION                               AVAILABLE   PROGRESSING   SINCE   STATUS
version   4.15.0-0.okd-scos-2024-01-18-223523   True        False         7h45m   Cluster version is 4.15.0-0.okd-scos-2024-01-18-223523

1. 
oc adm upgrade --to-image registry.ci.openshift.org/origin/release-scos:4.16.0-0.okd-scos-2024-07-21-033952 --force --allow-explicit-upgrade
warning: Using by-tag pull specs is dangerous, and while we still allow it in combination with --force for backward compatibility, it would be much safer to pass a by-digest pull spec instead
warning: The requested upgrade image is not one of the available updates. You have used --allow-explicit-upgrade for the update to proceed anyway
warning: --force overrides cluster verification of your supplied release image and waives any update precondition failures.
Requested update to release image registry.ci.openshift.org/origin/release-scos:4.16.0-0.okd-scos-2024-07-21-033952

2.
oc get clusterversion
NAME      VERSION                               AVAILABLE   PROGRESSING   SINCE   STATUS
version   4.15.0-0.okd-scos-2024-01-18-223523   True        True          3m27s   Unable to apply 4.16.0-0.okd-scos-2024-07-21-033952: some cluster configuration is invalid

3,
oc logs -n openshift-cluster-version cluster-version-operator-5987fd6b6-cfplw
E0722 23:34:06.714276       1 task.go:122] error running apply for customresourcedefinition "infrastructures.config.openshift.io" (47 of 895): CustomResourceDefinition.apiextensions.k8s.io "infrastructures.config.openshift.io" is invalid: [spec.validation.openAPIV3Schema.properties[spec].properties[platformSpec].properties[openstack].properties[machineNetworks].items.x-kubernetes-validations[0].rule: Invalid value: apiextensions.ValidationRule{Rule:"isCIDR(self)", Message:"value must be a valid CIDR network address", MessageExpression:"", Reason:(*apiextensions.FieldValueErrorReason)(nil), FieldPath:""}: compilation failed: ERROR: <input>:1:7: undeclared reference to 'isCIDR' (in container '')
| isCIDR(self)
| ......^, spec.validation.openAPIV3Schema.properties[spec].properties[platformSpec].properties[openstack].properties[apiServerInternalIPs].items.x-kubernetes-validations[0].rule: Invalid value: apiextensions.ValidationRule{Rule:"isIP(self)", Message:"value must be a valid IP address", MessageExpression:"", Reason:(*apiextensions.FieldValueErrorReason)(nil), FieldPath:""}: compilation failed: ERROR: <input>:1:5: undeclared reference to 'isIP' (in container '')
| isIP(self)
| ....^, spec.validation.openAPIV3Schema.properties[spec].properties[platformSpec].properties[openstack].properties[apiServerInternalIPs].x-kubernetes-validations[0].rule: Invalid value: apiextensions.ValidationRule{Rule:"size(self) == 2 && isIP(self[0]) && isIP(self[1]) ? ip(self[0]).family() != ip(self[1]).family() : true", Message:"apiServerInternalIPs must contain at most one IPv4 address and at most one IPv6 address", MessageExpression:"", Reason:(*apiextensions.FieldValueErrorReason)(nil), FieldPath:""}: compilation failed: ERROR: <input>:1:24: undeclared reference to 'isIP' (in container '')
| size(self) == 2 && isIP(self[0]) && isIP(self[1]) ? ip(self[0]).family() != ip(self[1]).family() : true
| .......................^

4. 
oc scale --replicas=0 deployments/cluster-version-operator -n openshift-cluster-version
deployment.apps/cluster-version-operator scaled
oc get pods -n openshift-cluster-version
NAME                   READY   STATUS      RESTARTS   AGE
version--pk2hp-l6tc6   0/1     Completed   0          5m23s

5. 
oc adm release info --image-for=hyperkube registry.ci.openshift.org/origin/release-scos:4.16.0-0.okd-scos-2024-07-21-033952
registry.ci.openshift.org/origin/4.16-okd-scos-2024-07-21-033952@sha256:2800d1392d9ef7759df244a709337d288ead95e75f439749f9fe57c347afe8b1

6. 
oc adm release info --image-for=cluster-kube-apiserver-operator registry.ci.openshift.org/origin/release-scos:4.16.0-0.okd-scos-2024-07-21-033952
registry.ci.openshift.org/origin/4.16-okd-scos-2024-07-21-033952@sha256:50194105e197b103e1af59efda006ed680f2e631c841d63abda7bf5444dcfca3

7. 
oc edit -n openshift-kube-apiserver-operator deployments/kube-apiserver-operator

  env:
    - name: IMAGE   
      value: registry.ci.openshift.org/origin/4.16-okd-scos-2024-07-21-033952@sha256:2800d1392d9ef7759df244a709337d288ead95e75f439749f9fe57c347afe8b1
    - name: OPERATOR_IMAGE
      value: registry.ci.openshift.org/origin/4.16-okd-scos-2024-07-21-033952@sha256:50194105e197b103e1af59efda006ed680f2e631c841d63abda7bf5444dcfca3
    - name: OPERAND_IMAGE_VERSION
      value: 1.29.6
    - name: OPERATOR_IMAGE_VERSION
      value: 4.15.0-0.okd-scos-2024-01-18-223523
    - name: POD_NAME
      valueFrom:
        fieldRef:
          apiVersion: v1
          fieldPath: metadata.name
    image: registry.ci.openshift.org/origin/4.16-okd-scos-2024-07-21-033952@sha256:50194105e197b103e1af59efda006ed680f2e631c841d63abda7bf5444dcfca3

8.
oc get pods -n openshift-kube-apiserver-operator && oc get pods -n openshift-kube-apiserver | grep -v Completed
NAME                                       READY   STATUS    RESTARTS   AGE
kube-apiserver-operator-5dd99f6bd6-cv4bn   1/1     Running   0          3m31s
NAME                          READY   STATUS      RESTARTS   AGE
kube-apiserver-okd-sno        5/5     Running     0          78s

9.
oc scale --replicas=1 deployments/cluster-version-operator -n openshift-cluster-version

10.
oc get clusterversion
NAME      VERSION                               AVAILABLE   PROGRESSING   SINCE   STATUS
version   4.16.0-0.okd-scos-2024-07-21-033952   True        False         8h      Cluster version is 4.16.0-0.okd-scos-2024-07-21-033952

oc get nodes -o wide
NAME      STATUS   ROLES                         AGE   VERSION           INTERNAL-IP    EXTERNAL-IP   OS-IMAGE                                    KERNEL-VERSION          CONTAINER-RUNTIME
okd-sno   Ready    control-plane,master,worker   20h   v1.29.6+aba1e8d   192.168.5.71   <none>        CentOS Stream CoreOS 417.9.202407210037-0   5.14.0-480.el9.x86_64   cri-o://1.29.5

oc get co
NAME                                       VERSION                               AVAILABLE   PROGRESSING   DEGRADED   SINCE   MESSAGE
authentication                             4.16.0-0.okd-scos-2024-07-21-033952   True        False         False      111m
baremetal                                  4.16.0-0.okd-scos-2024-07-21-033952   True        False         False      20h
cloud-controller-manager                   4.16.0-0.okd-scos-2024-07-21-033952   True        False         False      20h
cloud-credential                           4.16.0-0.okd-scos-2024-07-21-033952   True        False         False      20h
cluster-autoscaler                         4.16.0-0.okd-scos-2024-07-21-033952   True        False         False      20h
config-operator                            4.16.0-0.okd-scos-2024-07-21-033952   True        False         False      20h
console                                    4.16.0-0.okd-scos-2024-07-21-033952   True        False         False      9h
control-plane-machine-set                  4.16.0-0.okd-scos-2024-07-21-033952   True        False         False      20h
csi-snapshot-controller                    4.16.0-0.okd-scos-2024-07-21-033952   True        False         False      9h
dns                                        4.16.0-0.okd-scos-2024-07-21-033952   True        False         False      8h
etcd                                       4.16.0-0.okd-scos-2024-07-21-033952   True        False         False      20h
image-registry                             4.16.0-0.okd-scos-2024-07-21-033952   True        False         False      9h
ingress                                    4.16.0-0.okd-scos-2024-07-21-033952   True        False         False      20h
insights                                   4.16.0-0.okd-scos-2024-07-21-033952   True        False         False      20h
kube-apiserver                             4.16.0-0.okd-scos-2024-07-21-033952   True        False         False      20h
kube-controller-manager                    4.16.0-0.okd-scos-2024-07-21-033952   True        False         False      20h
kube-scheduler                             4.16.0-0.okd-scos-2024-07-21-033952   True        False         False      20h
kube-storage-version-migrator              4.16.0-0.okd-scos-2024-07-21-033952   True        False         False      20h
machine-api                                4.16.0-0.okd-scos-2024-07-21-033952   True        False         False      20h
machine-approver                           4.16.0-0.okd-scos-2024-07-21-033952   True        False         False      20h
machine-config                             4.16.0-0.okd-scos-2024-07-21-033952   True        False         False      20h
marketplace                                4.16.0-0.okd-scos-2024-07-21-033952   True        False         False      20h
monitoring                                 4.16.0-0.okd-scos-2024-07-21-033952   True        False         False      9h
network                                    4.16.0-0.okd-scos-2024-07-21-033952   True        False         False      20h
node-tuning                                4.16.0-0.okd-scos-2024-07-21-033952   True        False         False      9h
openshift-apiserver                        4.16.0-0.okd-scos-2024-07-21-033952   True        False         False      151m
openshift-controller-manager               4.16.0-0.okd-scos-2024-07-21-033952   True        False         False      115m
openshift-samples                          4.16.0-0.okd-scos-2024-07-21-033952   True        False         False      9h
operator-lifecycle-manager                 4.16.0-0.okd-scos-2024-07-21-033952   True        False         False      20h
operator-lifecycle-manager-catalog         4.16.0-0.okd-scos-2024-07-21-033952   True        False         False      20h
operator-lifecycle-manager-packageserver   4.16.0-0.okd-scos-2024-07-21-033952   True        False         False      20h
service-ca                                 4.16.0-0.okd-scos-2024-07-21-033952   True        False         False      20h
storage                                    4.16.0-0.okd-scos-2024-07-21-033952   True        False         False      20h

[titou10titou10]

So it's a success
Thanks

Will this fix be included in the next "Accepted" or "Stable" version for SCOS 4.16/4.17? Or the next stable v4.15 SCOS version?

[michael-willingham]

Has anyone gotten this guide to work on a single node cluster? After editing the kube-apiserver-operator, the actual kube-apiservers don't seem to update/pull in a new version. Re-scaling the cv operator leads to the same failure as before 😢

[madpearl]

You have to put the actual OKD build release in the "okd adm release info...", because old releases are deleted.

[michael-willingham]

Yeah, I've tried with the latest accepted / actually available releases, same issue sadly.

[michael-willingham]

For more clarity - I'm following this guide (for a SNO cluster - 4.15 fcos -> 4.15 scos -> 4.16 scos), and my kube-apiserver pods are never re-deploying with the updated image after modifying the kube-apiserver-operator. So I get the same CRD validation error in the cluster-version-operator. I started with a fresh 4.15 fcos install, and have done nothing previously besides the jump to 4.15 scos.

[aanoadb]

Did you succes to update from 4.15 okd to 4.15 okd-scos? any documentation how to update?

[titou10titou10]

@michael-willingham if you re-read my initial post and the various comments, you'll see that I've done it on a single node cluster (aka SNO)

[michael-willingham]

Thanks! Totally missed that this was a SNO cluster.

[titou10titou10]

Update from "stable" v4.15.0-0.okd-scos-2024-01-18-223523 to "next" v4.16.0-0.okd-scos-2024-07-29-184808 gives the same error ("'Could not update customresourcedefinition "infrastructures.config.openshift.io")
So the problem is still present in the latest SCOS versions

[madpearl]

I've upgraded to "next" 4.16.0-0.okd-scos-2024-08-01-132038 from stable 4.15.0-0.okd-scos-2024-01-18-223523 with the same procedure. I think it cannot be applied the openshift-kube-apiserver-operator config upgrade without going through all the steps of manual reconfiguration described from @aleskandro .

[jbparfum]

[GingerGeek]

Can you confirm what shape your OKD cluster is? Are you IPI/UPI? Single node/multi-node? Bare metal?

[Prashanth684]

could you log into the problematic node and get the logs from there? which systemd units are failing?

[tdemaret]

I have the same issue, maybe my logs can help. It is IPI deployment on Openstack :
NAME STATUS ROLES AGE VERSION INTERNAL-IP EXTERNAL-IP OS-IMAGE KERNEL-VERSION CONTAINER-RUNTIME
okd-8kb4s-master-0 Ready master 2y135d v1.29.8+f10c92d 172.32.1.111 10.10.11.187 CentOS Stream CoreOS 418.9.202409241032-0 5.14.0-511.el9.x86_64 cri-o://1.29.5
okd-8kb4s-master-1 Ready master 2y135d v1.29.8+f10c92d 172.32.2.18 CentOS Stream CoreOS 418.9.202409241032-0 5.14.0-511.el9.x86_64 cri-o://1.29.5
okd-8kb4s-master-2 Ready master 2y135d v1.29.8+f10c92d 172.32.2.126 CentOS Stream CoreOS 418.9.202409241032-0 5.14.0-511.el9.x86_64 cri-o://1.29.5
okd-8kb4s-worker-415-0-nz5np Ready worker 4d20h v1.28.7+6e2789b 172.32.1.165 Fedora CoreOS 39.20240210.3.0 6.7.4-200.fc39.x86_64 cri-o://1.28.2
okd-8kb4s-worker-415-0-svbj7 Ready,SchedulingDisabled worker 4d20h v1.29.8+f10c92d 172.32.1.173 CentOS Stream CoreOS 418.9.202409241032-0 5.14.0-511.el9.x86_64 cri-o://1.29.5
okd-8kb4s-worker-415-0-wgj46 Ready worker 4d17h v1.28.7+6e2789b 172.32.0.23 Fedora CoreOS 39.20240210.3.0 6.7.4-200.fc39.x86_64 cri-o://1.28.2
okd-8kb4s-worker-415-0-wlpxz Ready worker 4d17h v1.28.7+6e2789b 172.32.2.149 Fedora CoreOS 39.20240210.3.0 6.7.4-200.fc39.x86_64 cri-o://1.28.2

Any issue for masters, but it is not the same for worker node.
Uploading node-worker.log…

[jorgesantanaam]

@jbparfum i have the same issue as you. could you tell me how to solve it?

Followed the guide above, tried to upgrade from 4.15.0-0.okd-2024-03-10-010116 to 4.16.0-0.okd-scos-2024-09-27-110344
Machine config daemon crash looping with
I0930 12:57:49.876855 9599 start.go:68] Version: machine-config-daemon-4.6.0-202006240615.p0-2842-g6a425abb-dirty (6a425abbe5e330f0e7bc0849f4927d4c55d0c9e5) E0930 12:57:49.876997 9599 rpm-ostree.go:276] Merged secret file does not exist; defaulting to cluster pull secret I0930 12:57:49.877022 9599 rpm-ostree.go:263] Linking ostree authfile to /var/lib/kubelet/config.json F0930 12:57:49.942414 9599 start.go:106] Failed to initialize single run daemon: error reading osImageURL from rpm-ostree: exit status 1

[soubinan]

The fixes proposed here were not working for me
Actually, I was not so surprised to see some issues on the upgrade since it is a double change to deal with (lot of code changes + base OS change)
The easiest move (what I did) was to go from 4.15 okd to 4.15 okd-scos and then moving from the "new" 4.15 to 4.16
Sure it is double the work, but it is more reliable and without any workaround

[Elyytscha]

that are the exact version which worked for me, maybe this helps, i had to do #1971 (comment), at the switch to scos.

[soubinan]

For me, the apiserver never got updated after my changes on the operator
So the upgrade was looping on the error until the upgrade fail

[aanoadb]

Did you succes to update from 4.15 okd to 4.15 okd-scos? is there any documentation how to update?

[JOECHONG0420]

I can update from 4.15-0.okd-2024-03-10 to 4.16.0-okd-scos.1 successfully with the guideline at the top

[aanoadb]

@JOECHONG0420 your cluster vm uses BIOS or EFI firmware? if using BIOS firmware can it be directly updated without changing to EFI and disabling secure boot?

[jorgesantanaam]

Got the version 4.15 from upstream: https://amd64.origin.releases.ci.openshift.org/graph JOECHONG0420 ***@***.***> escreveu (quinta, 11/09/2025 à(s) 10:49):

…

default upstream of the graph? — Reply to this email directly, view it on GitHub <#1971 (reply in thread)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/BXGBEW76J7YIBCEWPJJ2M2T3SFAT7AVCNFSM6AAAAACFOQTJJOVHI2DSMVQWIX3LMV43URDJONRXK43TNFXW4Q3PNVWWK3TUHMYTIMZXGIZTSMA> . You are receiving this because you commented.Message ID: ***@***.***>