operator: multicluster end-to-end observability — raft metrics, reconcile-health, StretchCluster member status, PrometheusRule, dashboard

[hidalgopl]

Ships first-class observability for the multicluster operator. One PR, three metric families plus chart/dashboard/docs.

What's in

1. Multicluster raft metrics (operator_multicluster_raft_*)

The raft layer powering cross-cluster leader election was operationally invisible — only structured logs and a peer.dropCount atomic surfaced via a periodic logger goroutine. Diagnosing flapping leaders, slow peers, or chronic drops required eyeballing per-pod logs across all peers. New metrics, registered to controller-runtime's metrics registry so they ship out the operator's existing /metrics endpoint:

Push-based (incremented at the event site):

• leader_changes_total (Counter)
• messages_sent_total{msg_type, peer} / messages_received_total{msg_type, peer} — msg_type is a closed vocabulary of ~10 raft message types
• send_errors_total{peer, error_type} — error_type bucketed to timeout / canceled / unavailable / auth / marshal / other
• messages_dropped_total{peer} — replaces the deleted runDropLogger goroutine + peer.dropCount atomic
• send_duration_seconds{peer, result} — histogram with cross-region buckets (1ms..2.5s)
• inflight_rpcs{peer} / peer_reachable{peer} (Gauges)
• unreachable_reports_total{peer} / snapshots_sent_total{peer} / snapshot_send_errors_total{peer}
• follower_match_lag_entries{peer} — leader-only

Pull-based via prometheus.Collector (read from transport atomics on scrape, no hot-path writes):

• term (Gauge)
• state{state=...} — one series per state with 0/1 value, no separate is_leader gauge
• send_queue_length{peer} (Gauge)

The collector lives in pkg/multicluster/leaderelection/metrics.go, registered idempotently via RegisterTransport(t) after the transport's atomics are populated. Compile-time assertion (var _ prometheus.Collector = &transportCollector{}) ensures the interface stays implemented.

2. Reconcile-health metrics (operator_controller_*)

Wrapper-emitted automatically for every controller registered through observability.Wrap(reconciler, controller, defaultRequeueTimeout):

• reconcile_steady_state_total{controller} — Counter, incremented when a reconcile returns "no work to do" — either (Result{}, nil) or (Result{RequeueAfter: defaultRequeueTimeout}, nil) matching the controller's configured periodic-requeue interval. The second shape is required because MulticlusterReconciler always returns RequeueAfter = periodicRequeue via a defer; without the dual-shape predicate the StretchCluster controller would never register as steady.
• reconcile_last_success_timestamp_seconds{controller} — Gauge, Unix timestamp of the most recent steady-state reconcile. Prometheus computes "seconds since last success" at query time as time() - last_success_timestamp_seconds. Avoids the goroutine bookkeeping an imperative "seconds elapsed" gauge would need.

Wrap is generic over reconcile.TypedReconciler[R] so it covers both ctrl.Reconciler (single-cluster) and the multicluster reconciler (mcreconcile.Request) without duplicating the body.

Three controllers wrap: v2 Redpanda, NodePool, and StretchCluster. Two setup paths per controller — SetupWithManager (single-cluster binary, cmd/run) and SetupWithMultiClusterManager (multicluster binary, cmd/multicluster). Both paths wrap.

3. StretchCluster member-status metrics (operator_stretchcluster_*)

Per-member gauges, bounded cardinality (stretchcluster, member):

• member_reachable — 0/1 from the multicluster manager's reachability probe. Local cluster is always 1, recorded under its canonical name via lifecycle.CanonicalClusterName.
• brokers / brokers_ready — desired and ready broker counts per member, summed across NodePools pointing at that member. brokers - brokers_ready > 0 = partial outage.
• replication_health{stretchcluster} — 0/1, cluster-wide from the admin API health check reconcileDecommission already runs.
• spec_drift{stretchcluster, member} — 0/1, does each member's local StretchCluster.spec match the operator's view. Set inside checkSpecConsistency.

No new API calls — passive recorder, callers pass values they already have. MulticlusterReconciler is the only consumer; instrumented at three sites where the data already lives.

4. PrometheusRule (gated by monitoring.rulesEnabled chart value)

operator/chart/prometheusrule.go (transpiled to _prometheusrule.go.tpl by gotohelm). New monitoring.rulesEnabled is a sibling of monitoring.enabled (ServiceMonitor); independent so consumers can opt into rules without the ServiceMonitor.

Recording rules: operator:reconcile_rate:5m, operator:reconcile_error_rate:5m, operator:reconcile_steady_state_rate:5m, operator:reconcile_p99_seconds:5m.

Alerts, all severity=warning:

• OperatorReconcileErrors — operator:reconcile_error_rate:5m > 0.1 for 5m
• OperatorReconcileRunaway — operator:reconcile_rate:5m > 5 for 5m (the canonical "spinning controller" signal — cross-checks steady_state_total)
• OperatorReconcileStalled — active in the past hour but reconcile rate == 0 for 10m
• OperatorWorkerPoolSaturated — active_workers >= max_concurrent_reconciles for 10m
• StretchClusterMemberUnreachable — 2m
• StretchClusterBrokerCountSkew — 10m
• StretchClusterSpecDrift — 5m
• StretchClusterReplicationUnhealthy — 5m

5. Comprehensive Grafana dashboard (docs/operator-grafana-dashboard.json)

Single comprehensive dashboard, 5 rows: multicluster raft, StretchCluster member status, reconcile activity, queues & workers, reconcile-health signals.

Leader-only panels (Send latency p99, Follower match-lag entries) use and ignoring(...) to filter to the current leader's perspective — works in both dev-env (where remote_write adds a vcluster label) and prod (direct scrape with just instance).

6. Single source of truth for metric definitions

Every prometheus.New* call in the operator module now lives in operator/internal/observability/metrics.go (the v1 vectorized Cluster metrics in operator/internal/controller/vectorized/metric_controller.go are out of scope — v1 is legacy and explicitly not touched). Recorder helpers (RecordStretchCluster*) stay in stretch_recorder.go. The raft family stays in pkg/multicluster/leaderelection/metrics.go because that's a different Go module.

7. Documentation (docs/operator-metrics.md)

Canonical inventory of every metric the operator exposes. Cardinality table up front lists every label and its bounded vocabulary. Four groups: controller-runtime built-ins, reconcile-health, resource-state, multicluster raft.

8. Testing

• Unit tests (wrapper_test.go): all four record-path branches — Result{} on a controller with defaultRequeueTimeout=0, Result{RequeueAfter: defaultRequeueTimeout} (periodic-steady), Result{RequeueAfter: other} (real requeue, not steady), errors, immediate-requeue. Plus passthrough.
• Integration test (integration_test.go): TestIntegrationObservabilityInfiniteReconcile runs a synthetic reconciler inside a real controller-runtime Manager driven by envtest. Switches between spinning (RequeueAfter 100ms) and steady-state mid-test; asserts metrics react correctly. Gated by testutil.SkipIfNotIntegration + -tags integration.

Design notes worth flagging for reviewers

• leader_id is deliberately not a numeric gauge — sum(leader_id) is meaningless and state{state="leader"} == 1 already identifies the leader on each peer.
• state modelled as one series per state value (leader|follower|candidate|pre_candidate|unknown) with 0/1, not as a label-string-as-value. sum(state) == 1 invariant; state{state="leader"} == 1 is the standard leader filter.
• Wrapper's isPeriodicRequeue returns false when defaultRequeueTimeout == 0 so a stray Result{Requeue: true} (RequeueAfter == 0) on a non-periodic controller doesn't accidentally register as periodic-steady. Plain Result{} still counts via the result.IsZero() branch.