Feature: Throttling job creation

[1337andre]

We have a medium cluster with 20 nodes. When we try to cache e.g. CRD with 50 images kube-fledged will span 1000 jobs to pull that images. We can see that API server get in trouble and some workloads e.g. redis will have problems to get HA redis-cluster working. Does anyone have similar problems? Is it possible to throttling job creation?

[bkupidura]

I observed same behavior in my home lab. When bumping version of all my images (~21 managed by CRD), my kube api starts beeing unresponsible.

It would be nice if we can limit how many images can be downloaded in same time.

[senthilrch]

This is a much needed feature. Agreed.

[linuraj]

we tried throttling pod counts via ResourceQuota in the kube-fledged namespace. That didn't help either. Appreciate your help!

[senthilrch]

I've implented a solution for throttling jobs created by kube-fledged.
This will be delivered in v0.11.0 release

[aledeulo]

Hi, I would like to ask when you've planned to release the v0.11.0 version? Thank you very much.

[thomson131]

@senthilrch , firstly thank you for your work on this project, looking forward to the next release.

For those using this project, and waiting for the next release to better handle job throttling... a potential workaround is to apply both a pod quota and count/job.batch quota to the namespace. The operator seems to handle the job creation gracefully, waiting for job resources to become available and working through the imageCache list until completion within the resource constraints.