dark-vex · dark-vex · Aug 7, 2026 · Aug 7, 2026
@@ -279,20 +279,24 @@ jobs:
             done
           done
 
-      # k3s-rabbit's app inventory is exactly 3 apps (fluxcd,
-      # system-upgrade-controller, teleport-agent), and every one of them
-      # carries a real live external side-effect (Slack/GitHub
-      # notifications, a real k3s upgrade-channel call with node-cordon
-      # capability, a real Teleport tunnel join) — so EXCLUDED_APPS below
-      # matches 3/3 of them, and this loop deploys zero apps by design on
-      # every run, today. That's not a no-op: what this workflow still
+      # k3s-rabbit's app inventory is 4 apps (fluxcd, system-upgrade-controller,
+      # teleport-agent, flux-operator). 3 of them are excluded below —
+      # fluxcd, system-upgrade-controller, and teleport-agent each carry a
+      # real live external side-effect (Slack/GitHub notifications, a real
+      # k3s upgrade-channel call with node-cordon capability, a real
+      # Teleport tunnel join). flux-operator is NOT excluded and deploys
+      # for real via tier 3 below (its own per-app deploy.yaml) — same as
+      # oc-ampere's flux-operator app; its chart produces no Helm hooks and
+      # its Deployment idles until a FluxInstance CR exists, and
+      # flux-instance.yaml is never applied in this workflow. That's not a
+      # no-op run even setting flux-operator aside: what this workflow also
       # proves is Flux bootstrap, GitRepository resolution against the PR
       # branch, the single teleport-charts HelmRepository source
       # reconciling, the top-level system-upgrade-controller Kustomization
       # (applied generically above, distinct from the excluded app-level
       # Plan CRs of the same base name) reaching Ready, and fail-closed
       # coverage that fires automatically — failing this job — the moment a
-      # 4th app is ever added without an EXCLUDED_APPS entry or a real
+      # 5th app is ever added without an EXCLUDED_APPS entry or a real
       # resolution path.
       #
       # The conditions that actually gate this job's exit code: Mechanism
@@ -425,9 +429,11 @@ jobs:
       # Purely informational — `flux get` exits non-zero when it finds zero
       # matching objects (confirmed live: "no HelmRelease objects found in
       # any namespace" failed this step before `|| true` was added here).
-      # On k3s-rabbit specifically, zero HelmReleases ever exist by design
-      # (3/3 real apps excluded, see the "Deploy infra baseline and changed
-      # apps" step above) — every run would otherwise fail here every time.
+      # This guard stays needed generically: most runs (any PR that doesn't
+      # touch flux-operator) still deploy zero HelmReleases on k3s-rabbit
+      # (the other 3 real apps are excluded, see the "Deploy infra baseline
+      # and changed apps" step above), and a run that does touch
+      # flux-operator — like this PR's own — has exactly one.
       - name: Check resource deployment status
         run: |
           flux get kustomizations -A || true

@@ -75,7 +75,7 @@ clusters/{cluster-name}/
 
 - **Flux Operator clusters** (`kubenuc`, `k3s-rabbit`, `k8s-vms-daniele`, `oc-ampere`): Renovate auto-PRs `flux-instance.yaml` version bumps (`spec.distribution.version`) — do not manually edit unless fixing a bootstrap issue.
 - **Legacy classic-bootstrap clusters** (`kubenuc-test`, `k3s-prod-test`): Renovate auto-PRs `flux-system/gotk-components.yaml` bumps — same rule, don't hand-edit outside a bootstrap fix.
-- **Flux Operator's own installation** (the `controlplaneio-fluxcd/flux-operator` binary itself, distinct from the `FluxInstance` CR it reconciles): on `kubenuc`, `k8s-vms-daniele`, and `oc-ampere` this is now HelmRelease-managed via `clusters/{cluster}/apps/flux-operator/` (`charts/flux-operator.yml` HelmRepository + per-app Kustomization/HelmRelease), Renovate-tracked like any other chart. `k3s-rabbit` is still on the original untracked manual `helm install` — same gap, not yet closed on that cluster.
+- **Flux Operator's own installation** (the `controlplaneio-fluxcd/flux-operator` binary itself, distinct from the `FluxInstance` CR it reconciles): on all four Flux Operator clusters (`kubenuc`, `k8s-vms-daniele`, `oc-ampere`, `k3s-rabbit`) this is now HelmRelease-managed via `clusters/{cluster}/apps/flux-operator/` (`charts/flux-operator.yml` HelmRepository + per-app Kustomization/HelmRelease), Renovate-tracked like any other chart. `k3s-rabbit` was added last (PRs #1799/#1804/#1805), closing the gap previously noted here — no cluster remains on the untracked manual install for this component.
 - **`system-upgrade-controller`** (Rancher's k3s auto-upgrade controller, driven by `Plan` CRs under `apps/system-upgrade-controller/`): on `oc-ampere` this is now GitOps-managed (backported to `kubenuc`, `k8s-vms-daniele`, and `k3s-rabbit` in a companion PR) via `clusters/{cluster}/system-upgrade-controller/` (vendored verbatim `crd.yaml` + `controller.yaml` from the matching `rancher/system-upgrade-controller` GitHub release) plus a top-level `clusters/{cluster}/system-upgrade-controller.yaml` Kustomization that `apps` `dependsOn` (health-checked on the controller `Deployment`). This is a **new cluster-scoped CRD/RBAC install**, kept out of `apps/` deliberately — same ownership-separation reasoning as `flux-operator`/`charts`/`cluster-vars` each owning a sibling top-level directory. Renovate's `customManagers` entry only bumps the `image:` tag in `controller.yaml` — a version bump PR does **not** update the CRD schema, RBAC, or ConfigMap; those must be manually re-vendored from the matching release (same convention as `flux-system/gotk-components.yaml`). **Image tags have converged to `v0.20.1` across all four clusters** — not a deliberate pin. Renovate's `customManagers` entry auto-bumped the `image:` tag in all four `controller.yaml` files (commit `9829dab7`, PR #1700, 2026-07-25), exactly the tag-only-bump behavior described above; this was never reconciled against the "pinned to `v0.19.2`" decision previously documented here, which is now stale. Only `oc-ampere`'s vendored `crd.yaml` actually originates from the `v0.20.1` release — `kubenuc`/`k8s-vms-daniele`/`k3s-rabbit` still carry a `v0.19.2`-vintage `crd.yaml` alongside the newer image tag, an unreconciled CRD schema-version skew (CRD only — `controller.yaml`'s Deployment/RBAC/ConfigMap are byte-identical across both versions aside from the vendoring header comment). Not yet known to have caused any issue. See Confluence for the full incident history and live-check details.
 - **Explicit `apps/kustomization.yaml`** (listing every app's nested Kustomization file, or its directory for apps with their own wrapper `kustomization.yaml`, instead of relying on kustomize-controller's recursive auto-discovery) is now in place on all four Flux Operator clusters (`oc-ampere`, `k3s-rabbit`, `k8s-vms-daniele`, `kubenuc`) — this closes a dual-ownership bug where an app's live objects were reconciled by both the flattened `apps` Kustomization and its own nested child Kustomization (hit in practice on `oc-ampere`'s and `k8s-vms-daniele`'s `system-upgrade-controller` `Plan` CRs, and on `k8s-vms-daniele`'s `awx/backup/backup.yml` CronJob, which had no nested Kustomization CR of its own and is now listed explicitly to avoid it silently dropping out of the build). See "Add a new application to a cluster" above: any new app on one of these clusters must be added to its `apps/kustomization.yaml`'s `resources:` list.
 - **App-owned `Namespace` objects moved out of `manifests/` to the app's top level** on `k8s-vms-daniele` and `kubenuc` (PRs #1707–#1710), listed directly in the cluster's root `apps/kustomization.yaml` so the root `apps` Kustomization owns/creates them. The old pattern — `namespace.y*ml` nested inside `manifests/`, applied only by the app's `manifests` Kustomization, which itself `dependsOn`s the app's `secrets` Kustomization — deadlocks a from-scratch bootstrap: `secrets` can't go `Ready` without the namespace, and the `manifests` Kustomization that would create it never runs while blocked on `secrets`. Invisible on an already-live cluster (the namespace already exists), real on a genuine rebuild. `apps` applies with no `dependsOn` on anything app-specific, so this ownership makes the namespace structurally guaranteed to exist before any per-app Kustomization — `manifests` or `secrets` — can even start reconciling. Safe to move without triggering pruning of the (already-live) namespace because every such manifest carries `kustomize.toolkit.fluxcd.io/prune: disabled`, which exempts the object from garbage collection regardless of which Kustomization currently or formerly owns it. `oc-ampere` and `k3s-rabbit` don't need this — every split-secrets app there already bundles its own `namespace.yml` inside `secrets/` instead, so no `dependsOn` cycle exists on those two clusters. See "Add a new application to a cluster" above for the convention going forward.
@@ -97,7 +97,7 @@ Live on `kubenuc`, `k8s-vms-daniele`, and `k3s-rabbit` via `clusters/{cluster}/a
 |---|---|---|
 | `validate-kubenuc.yml` | PR | Full `kubenuc` cluster E2E validation (2h timeout) |
 | `validate-k8s-vms.yml` | PR | `k8s-vms-daniele` cluster validation |
-| `validate-k3s-rabbit.yml` | PR | `k3s-rabbit` cluster E2E validation (1h timeout) — real k3s + Flux install + reconcile, but the per-app loop deploys **zero apps by design** (all 3 real apps carry a live external side-effect and are excluded); proves bootstrap, GitRepository resolution, the chart source, and the top-level `system-upgrade-controller` install |
+| `validate-k3s-rabbit.yml` | PR | `k3s-rabbit` cluster E2E validation (1h timeout) — real k3s + Flux install + reconcile; 3 of 4 real apps are excluded (each carries a live external side-effect), and `flux-operator` deploys for real via tier 3 (same as `oc-ampere`'s `flux-operator`); also proves bootstrap, GitRepository resolution, the chart source, and the top-level `system-upgrade-controller` install |
 | `validate-oc-ampere.yml` | PR | `oc-ampere` cluster E2E validation (1h timeout) — 2 of 4 real apps (`flux-operator`, `ngx-webhook`) deploy for real; runs on an x86_64 self-hosted runner against genuinely ARM64 production compute, so it validates manifest/reconciliation correctness, not ARM64 image availability |
 | `validate-apps-kustomization.yml` | PR/push touching `{cluster}/apps/**` | Static check: every `apps/` directory is referenced by the cluster's root `apps/kustomization.yaml`, and every listed entry resolves to a real file/directory |
 | `security-static-analysis.yml` | PR/push to `clusters/**` | KubeLinter static analysis + checkov across all cluster apps |

@@ -12,4 +12,4 @@ Terse pointers to decisions that used to be inline comments — full rationale n
 
 ## CI e2e coverage (`validate-k3s-rabbit.yml`)
 
-This cluster has exactly 3 real apps — `fluxcd`, `system-upgrade-controller`, `teleport-agent` — and **all 3 are excluded** from the e2e workflow's real-path app loop: each has a live external side-effect (Slack/GitHub notifications, a real k3s upgrade-channel call with node-cordon capability, a real Teleport tunnel join against production infra). The per-app loop therefore deploys **zero apps by design** on every run today. This is not a no-op run: it still proves Flux bootstrap, GitRepository resolution against the PR branch, the `teleport-charts` HelmRepository source reconciling, and the top-level `system-upgrade-controller` Kustomization (a separate, safe object from the excluded app-level `Plan` CRs of the same base name) reaching Ready — plus fail-closed coverage that fires automatically the moment a 4th app is ever added without an `EXCLUDED_APPS` entry or a real resolution path. See the workflow's own header comments for the exact conditions that gate its exit code (the `Check for failed reconciliations` step does not — its `jq` pipelines all end in `|| true`).
+This cluster has 4 real apps — `fluxcd`, `system-upgrade-controller`, `teleport-agent`, `flux-operator` (the last added via PRs #1799/#1804/#1805). 3 of them are **excluded** from the e2e workflow's real-path app loop: each has a live external side-effect (Slack/GitHub notifications, a real k3s upgrade-channel call with node-cordon capability, a real Teleport tunnel join against production infra). `flux-operator` is **not** excluded and deploys for real via tier 3 (its own `deploy.yaml`) — same classification as `flux-operator` on `oc-ampere`: the chart produces no Helm hooks and its Deployment idles until a `FluxInstance` CR exists, and this workflow never applies `flux-instance.yaml`. This is not a no-op run even setting `flux-operator` aside: it still proves Flux bootstrap, GitRepository resolution against the PR branch, the `teleport-charts` HelmRepository source reconciling, and the top-level `system-upgrade-controller` Kustomization (a separate, safe object from the excluded app-level `Plan` CRs of the same base name) reaching Ready — plus fail-closed coverage that fires automatically the moment a 5th app is ever added without an `EXCLUDED_APPS` entry or a real resolution path. See the workflow's own header comments for the exact conditions that gate its exit code (the `Check for failed reconciliations` step does not — its `jq` pipelines all end in `|| true`).
@@ -1,3 +1,12 @@
+# flux-operator is NOT in EXCLUDED_APPS in validate-k3s-rabbit.yml's e2e
+# workflow — it deploys via the real deploy.y*ml fallback tier (tier 3),
+# same as oc-ampere. This comment-only touch exists solely to put
+# flux-operator into DEPLOY_APPS (derived purely from `git diff` against
+# main) so this PR's own CI run actually exercises that real deploy for
+# the first time on k3s-rabbit — flux-operator landed on this cluster via
+# separate PRs (#1799, #1804, #1805) while the e2e workflow PR was still
+# rebasing on top of them, so its own diff never included this app and no
+# live run has exercised the tier-3 real-deploy path for it here yet.
 ---
 apiVersion: helm.toolkit.fluxcd.io/v2
 kind: HelmRelease

@@ -12,7 +12,7 @@ Production `oc-ampere` runs on genuinely ARM64 compute — `terraform/oci/k8s-ar
 
 No `apps/1password/` and no 1Password Operator on this cluster (`FluxInstance.spec.components` only lists the four core Flux controllers) — this is a real gap, not a design choice scoped to a smaller app set. Any future app needing a `OnePasswordItem` secret needs that gap closed first (see `clusters/CLAUDE.md`'s Slack-alerting section for the same constraint).
 
-`flux-operator`'s own installation is HelmRelease-managed here (`apps/flux-operator/`, chart pinned to `0.57.0` via `charts/flux-operator.yml`), Renovate-tracked like any other chart — this cluster is not on the legacy manual `helm install` k3s-rabbit still uses.
+`flux-operator`'s own installation is HelmRelease-managed here (`apps/flux-operator/`, chart pinned to `0.57.0` via `charts/flux-operator.yml`), Renovate-tracked like any other chart — same pattern now in place on all four Flux Operator clusters, including `k3s-rabbit` (added via PRs #1799/#1804/#1805).
 
 ## CI e2e coverage (`validate-oc-ampere.yml`)
Original file line number	Diff line number	Diff line change
Expand Up		@@ -12,4 +12,4 @@ Terse pointers to decisions that used to be inline comments — full rationale n

		## CI e2e coverage (`validate-k3s-rabbit.yml`)

		This cluster has exactly 3 real apps — `fluxcd`, `system-upgrade-controller`, `teleport-agent` — and all 3 are excluded from the e2e workflow's real-path app loop: each has a live external side-effect (Slack/GitHub notifications, a real k3s upgrade-channel call with node-cordon capability, a real Teleport tunnel join against production infra). The per-app loop therefore deploys zero apps by design on every run today. This is not a no-op run: it still proves Flux bootstrap, GitRepository resolution against the PR branch, the `teleport-charts` HelmRepository source reconciling, and the top-level `system-upgrade-controller` Kustomization (a separate, safe object from the excluded app-level `Plan` CRs of the same base name) reaching Ready — plus fail-closed coverage that fires automatically the moment a 4th app is ever added without an `EXCLUDED_APPS` entry or a real resolution path. See the workflow's own header comments for the exact conditions that gate its exit code (the `Check for failed reconciliations` step does not — its `jq` pipelines all end in `\|\| true`).
		This cluster has 4 real apps — `fluxcd`, `system-upgrade-controller`, `teleport-agent`, `flux-operator` (the last added via PRs #1799/#1804/#1805). 3 of them are excluded from the e2e workflow's real-path app loop: each has a live external side-effect (Slack/GitHub notifications, a real k3s upgrade-channel call with node-cordon capability, a real Teleport tunnel join against production infra). `flux-operator` is not excluded and deploys for real via tier 3 (its own `deploy.yaml`) — same classification as `flux-operator` on `oc-ampere`: the chart produces no Helm hooks and its Deployment idles until a `FluxInstance` CR exists, and this workflow never applies `flux-instance.yaml`. This is not a no-op run even setting `flux-operator` aside: it still proves Flux bootstrap, GitRepository resolution against the PR branch, the `teleport-charts` HelmRepository source reconciling, and the top-level `system-upgrade-controller` Kustomization (a separate, safe object from the excluded app-level `Plan` CRs of the same base name) reaching Ready — plus fail-closed coverage that fires automatically the moment a 5th app is ever added without an `EXCLUDED_APPS` entry or a real resolution path. See the workflow's own header comments for the exact conditions that gate its exit code (the `Check for failed reconciliations` step does not — its `jq` pipelines all end in `\|\| true`).