http rate limit parameters not working properly

[zach593]

What happened:

Currently, there are 4 cmdline parameters for http rate limiting for all karmada components: cluster-api-qps, cluster-api-burst, kube-api-qps, and kube-api-burst. However, the limits of these parameters are not strict and sometimes do not work at all.

What you expected to happen:

The request qps of the Karmada component should strictly follow the qps/burst parameter.

How to reproduce it (as minimally and precisely as possible):

Using karmada-controller-manager as an example, it could also work on karmada-agent

1. a karmada cluster with lots of workloads and visiable monitoring, clusters generated by hack/local-up-karmada/sh could also work.
2. disable leader-elect, to avoid interference caused by failure of election.
3. set concurrent-work-syncs, concurrent-resourcebinding-syncs, concurrent-resource-template-syncs to 100 or larger, set kube-api-qps, kube-api-burst to 20, set cluster-api-qps, cluster-api-burst to 5, and start karmada-controller-manager.

Now we can check the monitoring through promql: sum by (pod, host)(increase(rest_client_requests_total{namespace="karmada-system",pod=~"karmada-controller-manager.+"}[1m]))/60, we can see that the actual qps accessing the karmada cluster and the member cluster far exceeds the limit we set.

Anything else we need to know?:

There are two main reasons.

1. QPS/burst parameters are not passed to the related clients.

This affects controller-manager and agent.
The cluster-api-qps and cluster-api-burst parameters are only passed to cluster-status-controller, while objectwatcher, which has more requirements for qps/burst, is not limited by these two parameters. The actual limit of its qps are concurrent-work-syncs and kube-api-qps/kube-api-burst, which limit the speed of updating work objects.

2. When creating multiple clients in a component, even if the same rest.Config is used, their ratelimits are still independent, not shared.

This affects most of the karmada components. I won't list them one by one.

karmada/cmd/scheduler/app/scheduler.go

Lines 149 to 153 in ae2d1c1

	restConfig.QPS, restConfig.Burst = opts.KubeAPIQPS, opts.KubeAPIBurst
	dynamicClientSet := dynamic.NewForConfigOrDie(restConfig)
	karmadaClient := karmadaclientset.NewForConfigOrDie(restConfig)
	kubeClientSet := kubernetes.NewForConfigOrDie(restConfig)

In the code above, there are 3 clients with the same qps/limit, and the total qps/burst is 3 times the parameters we set.

The actual impact of this issue is more severe in the controller manager because it needs to update a large number of karmada objects and workloads, which also use different clients. Not only clients accessing the karmada cluster are affected, but also clients accessing the member clusters because clients are generated dynamically. In the objectwatcher, it even generates a client for each update operation.

Environment:

• Karmada version: current master
• kubectl-karmada or karmadactl version (the result of kubectl-karmada version or karmadactl version):
• Others:

[XiShanYongYe-Chang]

Thanks~
In favor of #6095
/assign @zach593