Cancel Messages Related To A Flow-Control

[sancar]

Problem:
Cancelling a message the Flow-control does not take affect immediately. Especially with the rate option. In the current implementation, with the rate, The messages are scheduled to fire in the future. Because of this implementation, the cancel is recorded CANCEL_REQUESTED for all these messages and only logged as CANCELLED when the schedule time hits.

Another big problem is that, when you want to fire a new message with the flow control, all the slots in the future is already taken by -soon to be cancelled- messages. This makes cancel effectively unusable.

Solution:
The flow control is rewritten to take the CANCEL into account sooner. And new messages does not need to wait for old slots to be opened.
Details:
With the new implementation, the messages are not scheduled to the future but put to a ordered wait list to be fired when the flow control limit allows. When a flow control is cancelled. Again all the messages in progress are marked as CANCEL REQUESTED.

The new messages are immediately put to wait list. When a flow control limit is opened and waitlist needs to be consumed, the CANCEL REQUESTED messages are all CANCELLED immediately. This allows, new messages in the wait list are fired immediately as well.