Responses do not arrive

[lukasBleibt]

Expected behavior

We still expect every message and repsponse to arrive correctly at the subscribed clients

Actual behavior

We have the problem that sometimes our service does not receive the responses to their requests anymore.
It always happens for the same service (which has the highest load), for this example I will refer to it as Service A. Service A receives a lot of messages via Websocket. Those requests are sent to another service with a response topic. At somepoint the responses do not arrive at ServiceA anymore. We can see in the logs that the other services still receive the requests and send the responses to the response topic. We can also see that the reponses arrive at the hiveMq-Broker. But Service B (while still sending requests) just do not receive those responses anymore.
We do not know what causes this issue but for some reasons the subscriptions of Service A are removed or get lost?
In order to fix this issue we need to restart Service A. After it booted, it subscribes to the topics again and receives all the messages. There are no changes done to the service other then restarting it, which indicates that we subscribe correctly. Also all the other services we use in our system still are able to receive and send messages.

To Reproduce

That is the funny part. We have no idea what causes the broker/Service to act like this. It mostly happend after a deployment when we receive a lot of requests at the same time. But it also appeared out of nowhere, without any warning. The issue always affects the same services. The whole system runs again after the restart of the service.

Steps

Reproducer code

Details

• Affected HiveMQ CE version(s): 2021.2 & 2024.3 (not tested other versions)
• Used JVM version: 17