feat(federation): FederatedCluster registry + fleet status rollup (#1234, #1235)

[Defilan]

Description

First slice of the multi-cluster federation epic (#630): register each edge site once on the datacenter, and surface fleet-wide inference health in one place. Status only in this slice - no Model artifacts, no inference traffic, no prompts cross clusters.

Design + decomposition: the epic is split into a foundation (this) plus four later slices (#1236 distribution, #1237 router, #1238 dispatch), tracked on #630.

What this adds

• FederatedCluster CRD (new federation.llmkube.dev/v1alpha1 group, cluster-scoped). Two-writer status by design: the edge writes the observed fields (lastHeartbeatTime, observedVersion, capacity, inference); the datacenter controller writes ONLY phase.
• Spoke-initiated connectivity. Edge clusters dial home (outbound only), so it works behind NAT/firewall for air-gapped factory-floor sites. The edge operator, in a config-gated --federation-role=edge mode, lists local Models/InferenceServices/node-GPU capacity and patches its OWN FederatedCluster status subresource on the datacenter over a per-site RBAC-scoped client. The datacenter never connects to edges.
• Datacenter controller sets phase from heartbeat staleness (Connected <= 3x interval, Stale <= 10x, else Unreachable).
• llmkube fleet register mints a per-site least-privilege ServiceAccount + ClusterRole (exactly: get/update/patch on federatedclusters/status and get on federatedclusters, both resourceNames-scoped to that one site - nothing broader) + token, and prints the edge-operator config.
• llmkube fleet status reads all FederatedClusters for a per-site + fleet-wide health table.

Two-writer safety (the load-bearing invariant)

The edge patch is a MergeFrom on a live-fetched datacenter object with phase never assigned, so the merge patch omits the phase key entirely and can never clobber it. The datacenter's Status().Update carries the resourceVersion, so a concurrent edge write causes a 409 + requeue rather than a silent revert. Verified end to end by the reviews, including an envtest that seeds phase=Connected, runs an edge tick, and asserts phase survives.

Default install is unchanged

--federation-role defaults to hub; a default install runs only the datacenter controller, which with no FederatedCluster CRs reconciles nothing. No federation side effects unless explicitly configured.

Type of Change

• New feature (non-breaking; behind a default-off flag)

How Has This Been Tested?

• Pure unit tests (phase-from-staleness table, status-summary counting incl. multi-vendor GPU sum, least-privilege RBAC shape asserted exactly, fleet-status golden table)
• Envtest: datacenter phase transitions (Connected/Stale/Unreachable); edge push reflects seeded counts and leaves phase untouched
• make check-helm-rbac green (federation rules covered); go build ./..., GOOS=linux golangci-lint ./... clean; make manifests generate + CRD chart sync

Follow-ups now folded in

• Live RBAC-403 envtest (done). A dedicated RBAC-enforcing envtest authenticates as the per-site scoped token (with the admin client cert stripped so the token is the only credential) and asserts a real 403: own-status patch succeeds, other-cluster status / list / own-spec patch are all Forbidden. It calls the shipped fleet register code, so it tracks production, and closes the design's "provably cannot write another cluster's status" acceptance item at runtime, not just by construction.
• Real token expiry surfaced (done). fleet register now prints the apiserver's returned ExpirationTimestamp and warns when the cluster capped the token lifetime below what was requested, so a short-lived token is visible rather than a silent early expiry.

Remaining follow-up (Minor, non-blocking)

• observedVersion reports dev until an operator-build ldflags target wires the version. Token rotation automation is a separate later slice.

Checklist

• Follows project style; self-reviewed; docs added (docs/federation/registry-and-status.md, sample)
• No new warnings; unit + envtest pass locally
• AI assistance disclosed: implemented via AI subagents under human orchestration (per-task implement + review, then a whole-branch review that verified the two-writer invariant and RBAC scoping). I directed the design and reviews and own this PR.

Assisted-by: Claude Code (design, implementation, and multi-stage review under human direction); I reviewed and own the final result.

Fixes #1234
Fixes #1235

[codecov]

Codecov Report

❌ Patch coverage is 46.37965% with 274 lines in your changes missing coverage. Please review.

Files with missing lines	Patch %	Lines
pkg/cli/fleet.go	66.79%	66 Missing and 21 partials ⚠️
api/federation/v1alpha1/zz_generated.deepcopy.go	0.00%	78 Missing ⚠️
cmd/main.go	0.00%	56 Missing ⚠️
internal/controller/federation_edge_controller.go	52.50%	33 Missing and 5 partials ⚠️
internal/controller/federatedcluster_controller.go	59.37%	9 Missing and 4 partials ⚠️
api/federation/v1alpha1/federatedcluster_types.go	0.00%	2 Missing ⚠️

📢 Thoughts on this report? Let us know!