controlplane pods not scraped in rke2

[mjnagel]

Environment

Device and OS: AWS ec2
App version: 0.33.1 (snapshot)
Kubernetes distro being used: RKE2
Other: Hardened with CIS profile

Steps to reproduce

Deploy uds-core on rke2 cluster.

Expected result

Prometheus is able to scrape all configured targets.

Actual Result

3 controlplane targets show as down, with errors of connection refused:

• etcd
• kube-controller-manager
• kube-scheduler

This may be partially related to network policy restrictions in my environment similar to this issue.

Severity/Priority

Low/Medium - unsure of the effect of these targets being down.

Additional Context

There is a resolved issue and write up in this issue in Big Bang. The TLDR there is that these services are only listening on localhost, so connections from Prometheus get dropped. Unclear if this is RKE2 specific or more common across other distros.

[mjnagel]

Noting that we have a "fix" for this in our CI I believe (cc @noahpb ) - we may just want to document this for core since it's not something we can entirely control?

[noahpb]

Required config changes here in ci rke2 cluster config.

[mjnagel]

Definition of done for this issue will be documentation of RKE2 configuration to properly scrape these targets. We should also note the way to enable the coredns netpol here.

These doc notes could be added to our prerequisites or distribution support page as they fit.