Handle large tenant update gracefully

[chmouel]

This is a follow up on the discussion happenning here :

https://chat.openshift.io/developers/pl/reegarnowty9iedkyqujgfcm1o

When there is a large tenant update all pods gets awaken, we need to be able to
handle this gracefully.

A workflow like this, tenant update (for jenkins pod) start :

• Disable idler for the tenant that is about to be updated via a REST api call to idler,
• Unidle this user (via oc command or a REST api call)
• Do the tenant update.
• Idle the pods
• Renable idler for this user via a REST api

I am not sure if it would help that much the load but at least the flow will be
more controlled, it sure would be better than guessing what's happening on the events received to idler.

Feel free to update this issue if you think of a better/improved solution,

[chmouel]

@MatousJobanek @hrishin @sthaha

[pbergene]

Idler needs to be running as a full cluster can't handle necessarily handle all pods awake at the same time. This has usually been handled by adding a delay to the tenant update, to ensure idling kicks in before the cluster becomes overloaded.

[chmouel]

@pbergene yep, just to clarify, it's only disabling idler operation for the user/tenant that is going to be updated, not the whole idler, (updated the issue)

[pbergene]

@chmouel Ah. Good :) I'll read next time

[chmouel]

This probaby is related: #232 and #83

But it would be good to keep this issue to discuss the strategy of how to handle large tenant update, just not disabling idler for a user

[hrishin]

Yes, it also related to #301 issue.

[hrishin]

@chmouel you closed it intentionally?
just wanted to confirm

[chmouel]

I closed most of my old created issues, we are not working on idler anymore so may as well,