docs(clusters): document k3s-rabbit/oc-ampere e2e coverage and fix stale oc-ampere description

clusters/CLAUDE.md

@@ -9,7 +9,7 @@
 | `k3s-prod-test` | k3s | Production-like test environment, independent manifests | Legacy classic bootstrap (`flux-system/`) |
 | `k3s-rabbit` | k3s | Rabbit server cluster | Flux Operator (`FluxInstance`) |
 | `k8s-vms-daniele` | VMs | Development cluster | Flux Operator (`FluxInstance`) |
-| `oc-ampere` | k3s (OCI, ARM/Ampere) | Teleport agent only | Flux Operator (`FluxInstance`) |
+| `oc-ampere` | k3s (OCI, ARM/Ampere) | Flux Operator + Teleport agent + system-upgrade-controller + ngx-webhook (4 real apps) | Flux Operator (`FluxInstance`) |
 
 ---
 
@@ -85,7 +85,7 @@ clusters/{cluster-name}/
 Live on `kubenuc`, `k8s-vms-daniele`, and `k3s-rabbit` via `clusters/{cluster}/apps/fluxcd/` (`notifications.yaml`: a `notification.toolkit.fluxcd.io` `Provider`+`Alert` pair, plus `secrets.yaml`/`secrets/slack-secret.yml` syncing a shared `OnePasswordItem` at `vaults/k8s_secrets/items/slack-url` — the same 1Password item across all three, not a per-cluster secret). `eventMetadata.cluster` in the `Alert` must be set to the real cluster name; the tuned `exclusionList` (transient DNS/socket noise) should be copied verbatim. `kubenuc`/`k8s-vms-daniele` additionally have a `github` `Receiver` for push-triggered reconcile and (`kubenuc` only) a `betterstack-bridge` `Provider`/`Alert` scoped to Nextcloud maintenance — neither is part of this pattern; don't copy them when extending alerting to a new cluster.
 
 **Not extended to the other 3 clusters, deliberately:**
-- **`oc-ampere`** has no 1Password Operator by design (`FluxInstance.spec.components` only lists the four core Flux controllers, and there's no `apps/1password/`) — adding Slack alerting here would need introducing 1Password onto a cluster that's intentionally scoped to Teleport-agent-only. Needs a product decision (add 1Password there, or use a different secret mechanism), not a copy-paste.
+- **`oc-ampere`** has no 1Password Operator (`FluxInstance.spec.components` only lists the four core Flux controllers, and there's no `apps/1password/`) — adding Slack alerting here would need introducing 1Password onto the cluster. This is independent of app count: `oc-ampere` has 4 real apps (`flux-operator`, `system-upgrade-controller`, `teleport-agent`, `ngx-webhook`; see `clusters/oc-ampere/CLAUDE.md`), not just Teleport — the no-1Password gap, not app scope, is what blocks this. Needs a product decision (add 1Password there, or use a different secret mechanism), not a copy-paste.
 - **`kubenuc-test` and `k3s-prod-test`** are the two legacy classic-bootstrap clusters (auto-discovery, no `apps/kustomization.yaml`). Every app there is either flat auto-discovered files or a nested `Kustomization` CR that cross-references another cluster's real content (see "Add a new application to a cluster" and `kubenuc-test`'s `overlays kubenuc manifests` convention) — there's no clean shape for notifications specifically: a flat copy would apply the `OnePasswordItem` with no `dependsOn` on the (untracked, out-of-band) 1Password operator; a nested `secrets` Kustomization pointed at its own `apps/fluxcd/secrets/` would be double-reconciled by auto-discovery (the exact class of bug fixed in PRs #1697–#1710); and cross-referencing the paired live cluster's whole `apps/fluxcd/` directory drags in resources that don't belong on a test cluster (`kubenuc`'s Nextcloud-specific `betterstack-bridge`, or a `webhook-ingress.yaml` that needs its own `postBuild.substituteFrom` to avoid applying a literal, invalid `${FLUX_WEBHOOK_HOST:=...}` hostname). Needs its own design, not a template copy.
 - **`k3s-prod-test`'s existing `apps/fluxcd/deploy.yaml`** already cross-references `clusters/k8s-vms-daniele/apps/fluxcd` today, despite this cluster being documented as having "independent manifests" (unlike `kubenuc-test`'s explicit "overlays" convention) — likely a copy-paste of `kubenuc-test`'s pattern rather than an intentional choice. Any Flux alert this fires on `k3s-prod-test` would carry `eventMetadata.cluster: "k8s-vms-daniele"`, which is wrong. Flagged here, not fixed — resolving it (independent content vs. an intentional, metadata-patched cross-reference) is a separate decision.
 
@@ -97,11 +97,15 @@ Live on `kubenuc`, `k8s-vms-daniele`, and `k3s-rabbit` via `clusters/{cluster}/a
 |---|---|---|
 | `validate-kubenuc.yml` | PR | Full `kubenuc` cluster E2E validation (2h timeout) |
 | `validate-k8s-vms.yml` | PR | `k8s-vms-daniele` cluster validation |
+| `validate-k3s-rabbit.yml` | PR | `k3s-rabbit` cluster E2E validation (1h timeout) — real k3s + Flux install + reconcile, but the per-app loop deploys **zero apps by design** (all 3 real apps carry a live external side-effect and are excluded); proves bootstrap, GitRepository resolution, the chart source, and the top-level `system-upgrade-controller` install |
+| `validate-oc-ampere.yml` | PR | `oc-ampere` cluster E2E validation (1h timeout) — 2 of 4 real apps (`flux-operator`, `ngx-webhook`) deploy for real; runs on an x86_64 self-hosted runner against genuinely ARM64 production compute, so it validates manifest/reconciliation correctness, not ARM64 image availability |
 | `validate-apps-kustomization.yml` | PR/push touching `{cluster}/apps/**` | Static check: every `apps/` directory is referenced by the cluster's root `apps/kustomization.yaml`, and every listed entry resolves to a real file/directory |
 | `security-static-analysis.yml` | PR/push to `clusters/**` | KubeLinter static analysis + checkov across all cluster apps |
 | `gitleaks.yml` | PR/push to `main` | Secret scanning |
 
-PR validation runs k3s + Flux CD with a 2-hour timeout. Robot Framework E2E tests via `tests/robot/robot-test-job.yaml`.
+`validate-kubenuc.yml` runs k3s + Flux CD with a 2-hour timeout and Robot Framework E2E tests via `tests/robot/robot-test-job.yaml`; `validate-k8s-vms.yml`, `validate-k3s-rabbit.yml`, and `validate-oc-ampere.yml` have a smaller step footprint (no cert-manager/ClusterIssuer, no 1Password, no CoreDNS override, no Robot Framework) — keyed to conditions those clusters don't have (no `Ingress`/`cert-manager` on any of the three, no `-test` mirror cluster or `apps/1password/` on `k3s-rabbit`/`oc-ampere`).
+
+All four cluster E2E workflows use a shared `EXCLUDED_APPS`-first, fail-closed per-app loop: an app not in `EXCLUDED_APPS` must resolve via a `-test` mirror directory, a real per-app `deploy.y*ml`, or (added for `k3s-rabbit`/`oc-ampere`) a `dirname()`-resolved raw-file entry in the root `apps/kustomization.yaml` — anything else fails the job rather than silently skipping. No self-hosted-runner `concurrency:` guard exists across these — confirmed unnecessary: both self-hosted runners (`gen8-runner`/Lugano, `psp-runner`/BGY) are distinct physical machines, each processes one job at a time by default, and neither workflow pins a specific runner label, so GitHub's own per-runner queuing already prevents two k3s installs from landing on the same machine.
 
 **`validate-kubenuc.yml` and `validate-k8s-vms.yml` still don't build the real root `apps/kustomization.yaml`:** both derive their app list from `git diff` path-parsing and create one synthetic `flux create kustomization app-<name>` CR per changed app, never building/applying a Kustomization pointed at the whole `clusters/{cluster}/apps` directory. Both also deploy exclusively from the `-test` cluster directories (`kubenuc-test`, `k3s-prod-test`), which don't have a root `apps/kustomization.yaml` at all — only the prod directories (`kubenuc`, `k8s-vms-daniele`, plus `k3s-rabbit`/`oc-ampere`) do. The specific failure mode that motivated this — an app directory silently un-owned or double-listed because the real `resources:` list was never checked — is closed: `validate-apps-kustomization.yml` statically validates every cluster's real `apps/kustomization.yaml` directly (orphaned app directories, stale/typo'd entries) on every PR touching its `apps/`, independent of what the two E2E workflows happen to exercise. It does not run `kustomize build` against the file, so it won't catch invalid YAML or duplicate resource entries within an otherwise-complete list — only orphans and dangling paths. See memory `project_kubenuc_e2e_bootstrap_gap.md` / `project_k8s_vms_e2e_ci_gap.md` for the E2E-workflow history this doesn't change.
 

clusters/k3s-rabbit/CLAUDE.md

@@ -9,3 +9,7 @@
 Terse pointers to decisions that used to be inline comments — full rationale now lives in Confluence.
 
 - `fluxcd` (notifications) — the `fluxcd-notifications` Alert deliberately omits `eventMetadata.region`, unlike kubenuc/k8s-vms-daniele. See [Confluence: FluxCD — Notification Region Omission (k3s-rabbit)](https://fastnetserv.atlassian.net/wiki/spaces/IT/pages/748322817).
+
+## CI e2e coverage (`validate-k3s-rabbit.yml`)
+
+This cluster has exactly 3 real apps — `fluxcd`, `system-upgrade-controller`, `teleport-agent` — and **all 3 are excluded** from the e2e workflow's real-path app loop: each has a live external side-effect (Slack/GitHub notifications, a real k3s upgrade-channel call with node-cordon capability, a real Teleport tunnel join against production infra). The per-app loop therefore deploys **zero apps by design** on every run today. This is not a no-op run: it still proves Flux bootstrap, GitRepository resolution against the PR branch, the `teleport-charts` HelmRepository source reconciling, and the top-level `system-upgrade-controller` Kustomization (a separate, safe object from the excluded app-level `Plan` CRs of the same base name) reaching Ready — plus fail-closed coverage that fires automatically the moment a 4th app is ever added without an `EXCLUDED_APPS` entry or a real resolution path. See the workflow's own header comments for the exact conditions that gate its exit code (the `Check for failed reconciliations` step does not — its `jq` pipelines all end in `|| true`).

clusters/k3s-rabbit/kustomization.yaml

@@ -1,5 +1,9 @@
 apiVersion: kustomize.config.k8s.io/v1beta1
 kind: Kustomization
+# .github/workflows/validate-k3s-rabbit.yml parses this resources: list directly
+# (anything not one of apps.yaml/charts.yaml/cluster-vars.yaml/flux-instance.yaml
+# gets kubectl apply + wait'd generically) — a new entry here needs no matching
+# workflow change to get e2e coverage, but must still resolve to a real file.
 resources:
   - apps.yaml
   - charts.yaml

clusters/oc-ampere/CLAUDE.md

@@ -0,0 +1,23 @@
+# oc-ampere — Cluster-Specific Notes
+
+## Architecture: ARM64 production, x86_64 CI
+
+Production `oc-ampere` runs on genuinely ARM64 compute — `terraform/oci/k8s-armchair/main.tf`, `shape = "VM.Standard.A1.Flex"` (OCI Ampere A1). The self-hosted GitHub Actions runner pool this repo's CI uses is not documented as any specific architecture anywhere in this repo — not confirmed x86_64 by any repo file, but there's also no evidence of ARM64 self-hosted capacity.
+
+**Practical effect**: `validate-oc-ampere.yml`'s e2e run validates manifest correctness and Flux reconciliation health — it does not validate that every image this cluster pulls actually has an arm64 variant. Images not covered by that validation: `nginxinc/nginx-unprivileged:1.31-alpine` (`ngx-webhook`), `ghcr.io/controlplaneio-fluxcd/flux-operator:v0.57.0` (`flux-operator`). If a future image pin only ships an amd64 tag, this workflow will pass while production fails to schedule the pod — check multi-arch support by hand for any new image added here.
+
+## App Inventory
+
+4 real apps under `apps/kustomization.yaml`: `flux-operator`, `ngx-webhook`, `system-upgrade-controller`, `teleport-agent`. This supersedes any older "Teleport agent only" characterization of this cluster — see `clusters/CLAUDE.md`'s cluster table.
+
+No `apps/1password/` and no 1Password Operator on this cluster (`FluxInstance.spec.components` only lists the four core Flux controllers) — this is a real gap, not a design choice scoped to a smaller app set. Any future app needing a `OnePasswordItem` secret needs that gap closed first (see `clusters/CLAUDE.md`'s Slack-alerting section for the same constraint).
+
+`flux-operator`'s own installation is HelmRelease-managed here (`apps/flux-operator/`, chart pinned to `0.57.0` via `charts/flux-operator.yml`), Renovate-tracked like any other chart — this cluster is not on the legacy manual `helm install` k3s-rabbit still uses.
+
+## CI e2e coverage (`validate-oc-ampere.yml`)
+
+2 of the 4 real apps deploy for real in e2e: `flux-operator` and `ngx-webhook`. `system-upgrade-controller` and `teleport-agent` are excluded (same live-external-side-effect reasons as k3s-rabbit's: real `upgrade.cattle.io` `Plan` CRDs with `cordon: true`, and a real Teleport tunnel join against production infra respectively).
+
+`ngx-webhook` has no per-app `deploy.yaml` — it's a bare `manifests/deploy.yml` entry directly in the root `apps/kustomization.yaml`. The e2e workflow resolves this via a dedicated fourth resolution tier (`dirname()` of the matched `apps/kustomization.yaml` entry), not the standard per-app `deploy.y*ml` fallback used everywhere else.
+
+`flux-instance.yaml` stays permanently skip-listed in this workflow's top-level-resource loop: the job bootstraps Flux via a plain `flux install`, and applying `flux-instance.yaml` for real would hand reconciliation to the workflow's own real-path `flux-operator` app, fighting the CLI-installed components mid-run. Verified (via `helm template` against the pinned `0.57.0` chart with empty values) that `flux-operator`'s own Deployment stays idle — no Helm hooks, no automatic action at pod startup — until a `FluxInstance` CR actually exists, so deploying it for real without one is safe.

clusters/oc-ampere/kustomization.yaml

@@ -1,5 +1,13 @@
 apiVersion: kustomize.config.k8s.io/v1beta1
 kind: Kustomization
+# .github/workflows/validate-oc-ampere.yml parses this resources: list directly
+# (anything not one of apps.yaml/charts.yaml/cluster-vars.yaml/flux-instance.yaml
+# gets kubectl apply + wait'd generically) — a new entry here needs no matching
+# workflow change to get e2e coverage, but must still resolve to a real file.
+# flux-instance.yaml itself MUST stay skip-listed there: that job bootstraps
+# Flux via plain `flux install`, and applying flux-instance.yaml for real would
+# hand reconciliation to the workflow's own real-path flux-operator app,
+# fighting the CLI-installed components mid-run.
 resources:
   - apps.yaml
   - charts.yaml