Race-condition while calculating the object state.

[w1ll-i-code]

Describe the bug

The function Checkable::ProcessCheckResult in the file lib/icinga/checkable-check.cpp locks and unlocks the Mutex on the Objects several times. If two check results come in at the same time (e.g. through a running check and the /v1/actions/process-check-result endpoint) it can happen that both update the object before they dispatch the events to the IDO and notifications, resulting in the second state being dispatched multiple times:

To Reproduce

This is used in production for a countdown to notify if no event came in in the last few minutes

1. Create a check that always returns a warning
2. Create a service that runs that check every 30 seconds (or less, for more events)
3. Send repeatedly check results for that service to the /v1/actions/process-check-result endpoint
4. After some time, there should occur this exact scenario
5. The chances can be improved with multiple services
6. The chances are improved after a restart, as multiple services need to rerun the check, while a retry logic for the checkresult endpoint will send the new state.
7. The occurrence can be detected by reading out the icinga:history:stream:state redis stream from icingadb, checking if an event was sent twice.

Expected behavior

I expect all state changes to trigger the appropriate events, even if they happened at the same time.

Screenshots

Your Environment

Include as many relevant details about the environment you experienced the problem in

• Version used (icinga2 --version): r2.14.3-1
• Operating System and version: Red Hat Enterprise Linux 8.10

[w1ll-i-code]

Screenshot from the IDO, but this also occurs on icingadb, as the bug happens before the events to those are dispatched.