Detect when a config sync is running?

[dgiesselbach]

We have the problem that when the config sync is running, all API requests against the master are unsuccessful. The config sync takes atm about 2 minutes.

If an icinga check in powershell runs during this time window, which has to execute API requests against the master, this will result in an error.

First question:
Is there a way to query via API whether a config sync is currently running or not? -> If we could query whether a config sync is running or not, we could adapt the icinga powershell scripts accordingly. That would help us enormously.

Second question:
Is it possible to trigger a pre and post script before/after the config sync?

We now have over 2000 hosts and over 43000 checks, I'm afraid that the config sync will "at some point" take 10 minutes or more and that we'll create "wild" incidents during this time. How do you generally handle the config sync in "large" environments?

[dgiesselbach]

Any feedback?

You can see that the topic is also important for others. Can you include this as a feature request so that the handling of the config sync does NOT block the API? Like a queuing or caching function?

[oxzi]

Thank you for creating this issue, and please excuse the silence in the meantime. For now, I can only briefly address your specific questions, but perhaps this will suffice.

Is there a way to query via API whether a config sync is currently running or not?

Reloading the configuration is no synchronization, it is more like a restart of Icinga 2 itself. So during the reload there is no clean state, making API requests unreliable.

What kind of behavior would you like to see? Delay API calls until Icinga 2 is alive again? As you wrote yourself, this may take some time.

Is it possible to trigger a pre and post script before/after the config sync?

AFAIK, no such functionality is currently implemented. What would you like to use it for?

We now have over 2000 hosts and over 43000 checks, I'm afraid that the config sync will "at some point" take 10 minutes or more and that we'll create "wild" incidents during this time. How do you generally handle the config sync in "large" environments?

This depends a lot on your configuration. As your setup grows, please try to avoid too much Icinga DSL magic, possibly including apply rules. Anything that needs to be evaluated in your configuration will slow down a reload and eventually get expensive.

[dgiesselbach]

What kind of behavior would you like to see? Delay API calls until Icinga 2 is alive again? As you wrote yourself, this may take some time.

Yes, please queue/cache the API calls and then process them when Icinga 2 is functional again

AFAIK, no such functionality is currently implemented. What would you like to use it for?

Up to now, it is not possible to check whether a config reload is currently taking place. With these pre/post scripts, we could create a file as a "workaround", for example; if it exists, a config reload is active.

It would of course be better if we could solve the "problem" with the first point "queue/cache".

This depends a lot on your configuration. As your setup grows, please try to avoid too much Icinga DSL magic, possibly including apply rules. Anything that needs to be evaluated in your configuration will slow down a reload and eventually get expensive.

Pass this on to my colleagues