You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
We're unlikely to get good resource utilization with CPU based autoscaling for workers, as it has not been overly effective for celery workers.
We'll need a way to have taskbrokers inform worker autoscaling. Ideally, as a broker instances's processing lag increases we scale up the worker pool. Conversely, when processing lag is low and task fetches start to miss we can scale worker pools down.
A potential solution is to build a prometheus style health/metrics endpoint into taskbroker that can be consumed by keda or k8s. We might be able to use the rabbitmq keda scaler integration as prior art as well.
The text was updated successfully, but these errors were encountered:
We're unlikely to get good resource utilization with CPU based autoscaling for workers, as it has not been overly effective for celery workers.
We'll need a way to have taskbrokers inform worker autoscaling. Ideally, as a broker instances's processing lag increases we scale up the worker pool. Conversely, when processing lag is low and task fetches start to miss we can scale worker pools down.
A potential solution is to build a prometheus style health/metrics endpoint into taskbroker that can be consumed by keda or k8s. We might be able to use the rabbitmq keda scaler integration as prior art as well.
The text was updated successfully, but these errors were encountered: