Add the way to scrape kubelet metrics via kube-apiserver network proxy.

[cndoit18]

What would you like to be added:

Add the way to scrape kubelet metrics via kube-apiserver network proxy.

Why is this needed:

The current kubelet scraping is not suitable for situations where the master network and the node network are isolated.

https://github.com/kubernetes/enhancements/tree/master/keps/sig-api-machinery/1281-network-proxy#proposal

/kind feature

[yangjunmyfm192085]

metrics-serverscrapes data from kubelet directly .
may be It is also an good option to support data fetching through apiserver network proxy.
I think it's worth , and it shouldn't affect performance.

[yangjunmyfm192085]

/assign

[serathius]

As it was discovered by Prometheus community, apiserver proxy was not designed to handle large traffic like metrics collection. I would be opposed to this idea if there are no other arguments then "it should do it". Even in the proxy design it mentioned that for performance proxy should be bypassed.

[cndoit18]

As it was discovered by Prometheus community, apiserver proxy was not designed to handle large traffic like metrics collection. I would be opposed to this idea if there are no other arguments then "it should do it". Even in the proxy design it mentioned that for performance proxy should be bypassed.

In our use case, we deploy both the metrics-server and HPA on the master, while the actual business logic runs on the node. However, these two networks are isolated, which prevents us from using the HPA. therefore, we want to add a functional switch that can be used to get the metrics through the tunnel.

[serathius]

This is a bad idea due to blast radius. Sharing konnectivity proxy between apiserver and proxy could result in metrics-server starving your apiserver connections. Metrics server traffic is very very very bursty and as it was not designed to work through proxy. It has high chance of interrupting your apiserver webhook execution and taking down whole cluster. I would strongly recommend to not bind reliability of critical components like apiserver with non-critical one like metrics-server.

/cc @mborsz

Fact that you can configure your K8s cluster like this doesn't mean you should.

[cndoit18]

This is a bad idea due to blast radius. Sharing konnectivity proxy between apiserver and proxy could result in metrics-server starving your apiserver connections. Metrics server traffic is very very very bursty and as it was not designed to work through proxy. It has high chance of interrupting your apiserver webhook execution and taking down whole cluster. I would strongly recommend to not bind reliability of critical components like apiserver with non-critical one like metrics-server.

/cc @mborsz

Fact that you can configure your K8s cluster like this doesn't mean you should.

Is it possible to explore an alternative solution by supporting a connection to konnectivity-server, rather than using kube-apiserver, to address this issue?

[serathius]

Is it possible to explore an alternative solution by supporting a connection to konnectivity-server, rather than using kube-apiserver, to address this issue?

As long as information on how to connect to Kubelet via konnectivity-server is published on Kubernetes Node object. This is how it currently works that Kubelet publishes it's address to Node object status and metrics server just reads it. It allows Kubelet to publish both InternalIP and externalIP and metrics server have a flag to pick which it should use.

[dashpole]

/triage accpeted

[k8s-ci-robot]

@dashpole: The label(s) triage/accpeted cannot be applied, because the repository doesn't have them.

In response to this:

/triage accpeted

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[dashpole]

/triage accepted