Improve ClusterCacheTracker to address fundamental limitations

[sbueringer]

Over time the ClusterCacheTracker evolved significantly. While it generally improved we have a few systematic issues that we should solve:

• the current locking and client creation behavior is not good, whenever a reconciler calls GetClient it has a chance to be stuck for 10 seconds if the wl cluster is unreachable (because then GetClient might tries to create a client and then times out after 10s). While one reconciler tries to create a client other reconcilers will permanently requeue (currently with requeueAfter 1m). See also Improve ClusterCacheTracker TryLock behavior #10819
• I think in general there is a huge potential to make the locking smarter
• the current health checking code starts 1 go routine for every cluster (so 1k clusters => 1k health check goroutines)
• ClusterCacheTracker requires to also add the ClusterCache reconcile to the manager. Because ClusterCache reconcile is a separate component that is easily forgotten

I probably missed a few :)

We also have some additional requirements:

• We want to expose some information about the health checking state so that controllers can tell when the connection to the wl cluster broke and e.g. set the RemoteConnectionProbe condition accordingly (xref: 📖 Proposal: Improving status in CAPI resources #10897)

Tasks:

• Implement new version of ClusterCacheTracker (called ClusterCache): ✨ Introduce new ClusterCache #11247

Follow-up Tasks:

• [P0] Add panic metric validation to envtest package so we have it everywhere: 🌱 Check for panics during test runs in envtest #11279
• [P0] Deprecate the old ClusterCacheTracker: 🌱 Deprecate ClusterCacheTracker #11312
• [P1] Remove requeues on "ErrClusterNotConnected" in other reconcilers
  • Run some scale e2e tests to verify (including checking some Go profiles while we're at it)
• [Cleanup] Input validation & get rid of r.ClusterCache == nil checks
  • Add input validation to all SetupWithManager funcs for stuff like Client, ClusterCache, SecretCachingClient: 🌱 Add input validations for controllers #11327
  • Get rid of r.ClusterCache == nil checks machine_controller.go & machinehealthcheck_controller.go: 🌱 Remove clustercache nil checks #11336
• ✨ add typed watcher to ClusterCache #11331
• Add metrics based on health checking data & connection state (previous issue: Add metrics for realtime investigation of disconnected wl scenarios #7147)

Backlog:

• Consider exposing and using ReaderFailOnMissingInformer cache option
• Consider more sophisticated backoff behavior for connection creation (unclear if necessary / wanted)
• Consider making ClusterCache a generic cache per cluster, e.g. for:
  • etcd client certs (to replace GetClientCertificatePrivateKey)
    • Potentially also just use one private key for KCP instead of one per-Cluster (we can also consider rotating this one key if necessary)
  • but also maybe other stuff like drain cache
  • Important! Must be safe against deadlocks!

[sbueringer]

/area clustercachetracker
/triage accepted