Fix cert-manager CA rotation race in TLS cert rotation KUTTL test

The ctlplane-tls-cert-rotation KUTTL test fails intermittently because
the custom_duration patch changes both CA and leaf cert durations
simultaneously. cert-manager processes Certificate resources in
parallel, so leaf certs can be re-issued before the CA itself is
re-issued, resulting in some certs signed by the old CA and others by
the new CA. This causes cross-service SSL verification failures (e.g.
neutron cannot connect to OVN NB due to CA mismatch).

Fix by removing CA duration changes from the patch so only leaf cert
durations change, preventing the CA key from rotating. Also add
cert-manager re-issuance waits and control plane stability checks in
step 03, and add retry logic to the non-API service cert check in
step 04.

Ref: https://redhat.atlassian.net/browse/OSPRH-32142

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

Author: abays

config/samples/tls/custom_duration/patch.yaml

@@ -5,23 +5,15 @@ metadata:
 spec:
   tls:
     ingress:
-      ca:
-        duration: 1000h0m0s
       cert:
         duration: 500h0m0s
     podLevel:
       internal:
-        ca:
-          duration: 1000h0m0s
         cert:
           duration:  500h0m0s
       libvirt:
-        ca:
-          duration: 1000h0m0s
         cert:
           duration:  500h0m0s
       ovn:
-        ca:
-          duration: 1000h0m0s
         cert:
           duration:  500h0m0s

test/kuttl/tests/ctlplane-tls-cert-rotation/03-assert-new-certs.yaml

@@ -19,6 +19,34 @@ metadata:
 spec:
   duration: 500h0m0s
 ---
+apiVersion: cert-manager.io/v1
+kind: Certificate
+metadata:
+  name: neutron-internal-svc
+spec:
+  duration: 500h0m0s
+---
+apiVersion: cert-manager.io/v1
+kind: Certificate
+metadata:
+  name: glance-default-internal-svc
+spec:
+  duration: 500h0m0s
+---
+apiVersion: cert-manager.io/v1
+kind: Certificate
+metadata:
+  name: cinder-internal-svc
+spec:
+  duration: 500h0m0s
+---
+apiVersion: cert-manager.io/v1
+kind: Certificate
+metadata:
+  name: placement-internal-svc
+spec:
+  duration: 500h0m0s
+---
 apiVersion: core.openstack.org/v1beta1
 kind: OpenStackControlPlane
 metadata:
@@ -196,25 +224,17 @@ spec:
   tls:
     ingress:
       enabled: true
-      ca:
-        duration: 1000h0m0s
       cert:
         duration: 500h0m0s
     podLevel:
       enabled: true
       internal:
-        ca:
-          duration: 1000h0m0s
         cert:
           duration:  500h0m0s
       libvirt:
-        ca:
-          duration: 1000h0m0s
         cert:
           duration:  500h0m0s
       ovn:
-        ca:
-          duration: 1000h0m0s
         cert:
           duration:  500h0m0s
 status:

test/kuttl/tests/ctlplane-tls-cert-rotation/04-assert-service-cert-rotation.yaml

@@ -2,8 +2,21 @@ apiVersion: kuttl.dev/v1beta1
 kind: TestAssert
 timeout: 900
 commands:
+  - script: |
+      echo "Waiting for OpenStack control plane to be ready after cert re-issuance..."
+      oc wait openstackcontrolplane -n $NAMESPACE --for=condition=Ready --timeout=600s -l core.openstack.org/openstackcontrolplane
+
   - script: |
       echo "Checking rotation of non API service certificates..."
+      for i in $(seq 1 15); do
+        if NAMESPACE=$NAMESPACE bash ../../common/osp_check_noapi_service_certs.sh 2>/dev/null; then
+          echo "Non-API service certificates verified successfully."
+          exit 0
+        fi
+        echo "Attempt $i/15: Non-API service certs not yet consistent, waiting 20s..."
+        sleep 20
+      done
+      echo "Final non-API service cert check..."
       NAMESPACE=$NAMESPACE bash ../../common/osp_check_noapi_service_certs.sh
 
   - script: |