Race-condition during object shutdown

[w1ll-i-code]

Describe the bug

The function Checkable::ProcessCheckResult in the file lib/icinga/checkable-check.cpp locks and unlocks the Mutex on the Objects several times.

If a checkresult comes in, while icinga2 is shutting down (and #10191 is already applied, otherwise it's indistinguishable from the other errors). The object will change state while the other objects are already shutting down, preventing the correct events to be dispatched, resulting in an inconstant state between the db and icinga2.

To Reproduce

1. Add a sleep after each unlock in the function.
2. Restart icinga2 repeatedly while having a lot of services that often change states.
3. The occurrence can be detected by reading out the icinga:history:stream:state redis stream from icingadb, checking if the sequence of state changes for each object make sense.

Expected behavior

I expect all the state changes to dispatch the corresponding events correctly, to have the correct behavior for notifications and a full state history in the database.

Your Environment

Include as many relevant details about the environment you experienced the problem in

• Version used (icinga2 --version): r2.14.3-1
• Operating System and version: Red Hat Enterprise Linux 8.10