Avoid CPU throttling alert on spark-observer (re)start

[bajtos]

When the spark-observer service is restarted, typically after we deploy a new version, it consumes too much CPU, which triggers the CPU throttling warning.

We already did some optimisations in #316 and #319, but it was not enough.

What else can we try to avoid a false-positive CPU throttling alert at service startup?

[bajtos]

Interestingly enough, both observer and stats services trigger the CPU throttling alert at the same time 🤔

[juliangruber]

What else can we try to avoid a false-positive CPU throttling alert at service startup?

How do you know it's a false positive?

[bajtos]

You are correct, it's not a false positive.

What I wanted to write:

What else can we try to avoid a CPU throttling alert at service startup? Such alerts are not actionable for us. I am concerned that we will eventually learn to ignore all CPU throttling alerts from spark-observer and miss a rare alert that needs to be acted upon.

[juliangruber]

I see three options:

1. Try to make operations more efficient (looks like that has been tried)
2. Perform less operations per time, so as not to exceed the CPU quota
3. Provision more CPUs