Queue size exceeded, waiting for messages to be processed by paho client.

[franklupo]

Describe the issue
Through the gateway I read devices in MODBUS every 15 minutes and send to the cloud.
There were network problems and the communication stopped working.
The network started working again but it was not transmitting anymore. I had to restart the docker service and lost all the data.

Error traceback (If it was raised):

2025-01-06 22:40:21 - |WARNING| - [tb_device_mqtt.py] - tb_device_mqtt - __send_split_message - 925 - Queue size exceeded, waiting for messages to be processed by paho client.
2025-01-06 22:40:21 - |WARNING| - [tb_device_mqtt.py] - tb_device_mqtt - __send_split_message - 925 - Queue size exceeded, waiting for messages to be processed by paho client.
2025-01-06 22:40:32 - |WARNING| - [tb_device_mqtt.py] - tb_device_mqtt - __send_split_message - 925 - Queue size exceeded, waiting for messages to be processed by paho client.
2025-01-06 22:40:32 - |WARNING| - [tb_device_mqtt.py] - tb_device_mqtt - __send_split_message - 925 - Queue size exceeded, waiting for messages to be processed by paho client.
2025-01-06 22:40:32 - |WARNING| - [tb_device_mqtt.py] - tb_device_mqtt - __send_split_message - 925 - Queue size exceeded, waiting for messages to be processed by paho client.
2025-01-06 22:40:43 - |WARNING| - [tb_device_mqtt.py] - tb_device_mqtt - __send_split_message - 925 - Queue size exceeded, waiting for messages to be processed by paho client.
2025-01-06 22:40:43 - |WARNING| - [tb_device_mqtt.py] - tb_device_mqtt - __send_split_message - 925 - Queue size exceeded, waiting for messages to be processed by paho client.
2025-01-06 22:40:43 - |WARNING| - [tb_device_mqtt.py] - tb_device_mqtt - __send_split_message - 925 - Queue size exceeded, waiting for messages to be processed by paho client.
2025-01-06 22:40:43 - |WARNING| - [tb_device_mqtt.py] - tb_device_mqtt - __send_split_message - 925 - Queue size exceeded, waiting for messages to be processed by paho client.
2025-01-06 22:40:53 - |INFO| - [slave.py] - slave - connect - 166 - Connected to IT50 - Pressa 25
2025-01-06 22:40:53 - |ERROR| - [modbus_connector.py] - modbus_connector - __poll_device - 239 - Socket is closed, connection is lost, for device IT50 - Pressa 25
2025-01-06 22:40:54 - |WARNING| - [tb_device_mqtt.py] - tb_device_mqtt - __send_split_message - 925 - Queue size exceeded, waiting for messages to be processed by paho client.
2025-01-06 22:40:54 - |WARNING| - [tb_device_mqtt.py] - tb_device_mqtt - __send_split_message - 925 - Queue size exceeded, waiting for messages to be processed by paho client.
2025-01-06 22:40:54 - |WARNING| - [tb_device_mqtt.py] - tb_device_mqtt - __send_split_message - 925 - Queue size exceeded, waiting for messages to be processed by paho client.
2025-01-06 22:40:54 - |WARNING| - [tb_device_mqtt.py] - tb_device_mqtt - __send_split_message - 925 - Queue size exceeded, waiting for messages to be processed by paho client.
2025-01-06 22:40:59 - |INFO| - [slave.py] - slave - connect - 166 - Connected to IT60 - Macchina 46
2025-01-06 22:40:59 - |ERROR| - [modbus_connector.py] - modbus_connector - __poll_device - 239 - Socket is closed, connection is lost, for device IT60 - Macchina 46
2025-01-06 22:41:05 - |WARNING| - [tb_device_mqtt.py] - tb_device_mqtt - __send_split_message - 925 - Queue size exceeded, waiting for messages to be processed by paho client.
2025-01-06 22:41:16 - |WARNING| - [tb_device_mqtt.py] - tb_device_mqtt - __send_split_message - 925 - Queue size exceeded, waiting for messages to be processed by paho client.
2025-01-06 22:41:21 - |INFO| - [slave.py] - slave - connect - 166 - Connected to IT60 - Uffici Piano Terra
2025-01-06 22:41:21 - |ERROR| - [modbus_connector.py] - modbus_connector - __poll_device - 239 - Socket is closed, connection is lost, for device IT60 - Uffici Piano Terra
2025-01-06 22:41:27 - |WARNING| - [tb_device_mqtt.py] - tb_device_mqtt - __send_split_message - 925 - Queue size exceeded, waiting for messages to be processed by paho client.
2025-01-06 22:41:27 - |WARNING| - [tb_device_mqtt.py] - tb_device_mqtt - __send_split_message - 925 - Queue size exceeded, waiting for messages to be processed by paho client.
2025-01-06 22:41:27 - |WARNING| - [tb_device_mqtt.py] - tb_device_mqtt - __send_split_message - 925 - Queue size exceeded, waiting for messages to be processed by paho client.
2025-01-06 22:41:38 - |WARNING| - [tb_device_mqtt.py] - tb_device_mqtt - __send_split_message - 925 - Queue size exceeded, waiting for messages to be processed by paho client.
2025-01-06 22:41:38 - |WARNING| - [tb_device_mqtt.py] - tb_device_mqtt - __send_split_message - 925 - Queue size exceeded, waiting for messages to be processed by paho client.
2025-01-06 22:41:38 - |WARNING| - [tb_device_mqtt.py] - tb_device_mqtt - __send_split_message - 925 - Queue size exceeded, waiting for messages to be processed by paho client.
2025-01-06 22:41:49 - |WARNING| - [tb_device_mqtt.py] - tb_device_mqtt - __send_split_message - 925 - Queue size exceeded, waiting for messages to be processed by paho client.
2025-01-06 22:41:49 - |WARNING| - [tb_device_mqtt.py] - tb_device_mqtt - __send_split_message - 925 - Queue size exceeded, waiting for messages to be processed by paho client.
2025-01-06 22:41:49 - |WARNING| - [tb_device_mqtt.py] - tb_device_mqtt - __send_split_message - 925 - Queue size exceeded, waiting for messages to be processed by paho client.
2025-01-06 22:41:49 - |WARNING| - [tb_device_mqtt.py] - tb_device_mqtt - __send_split_message - 925 - Queue size exceeded, waiting for messages to be processed by paho client.
2025-01-06 22:41:55 - |INFO| - [slave.py] - slave - connect - 166 - Connected to IT50 - Pressa 24
2025-01-06 22:41:55 - |ERROR| - [modbus_connector.py] - modbus_connector - __poll_device - 239 - Socket is closed, connection is lost, for device IT50 - Pressa 24
2025-01-06 22:42:00 - |WARNING| - [tb_device_mqtt.py] - tb_device_mqtt - __send_split_message - 925 - Queue size exceeded, waiting for messages to be processed by paho client.
2025-01-06 22:42:11 - |WARNING| - [tb_device_mqtt.py] - tb_device_mqtt - __send_split_message - 925 - Queue size exceeded, waiting for messages to be processed by paho client.
2025-01-06 22:42:11 - |WARNING| - [tb_device_mqtt.py] - tb_device_mqtt - __send_split_message - 925 - Queue size exceeded, waiting for messages to be processed by paho client.
2025-01-06 22:42:22 - |WARNING| - [tb_device_mqtt.py] - tb_device_mqtt - __send_split_message - 925 - Queue size exceeded, waiting for messages to be processed by paho client.
2025-01-06 22:42:22 - |WARNING| - [tb_device_mqtt.py] - tb_device_mqtt - __send_split_message - 925 - Queue size exceeded, waiting for messages to be processed by paho client.
2025-01-06 22:42:22 - |WARNING| - [tb_device_mqtt.py] - tb_device_mqtt - __send_split_message - 925 - Queue size exceeded, waiting for messages to be processed by paho client.
2025-01-06 22:42:33 - |WARNING| - [tb_device_mqtt.py] - tb_device_mqtt - __send_split_message - 925 - Queue size exceeded, waiting for messages to be processed by paho client.
2025-01-06 22:42:44 - |WARNING| - [tb_device_mqtt.py] - tb_device_mqtt - __send_split_message - 925 - Queue size exceeded, waiting for messages to be processed by paho client.
2025-01-06 22:42:55 - |WARNING| - [tb_device_mqtt.py] - tb_device_mqtt - __send_split_message - 925 - Queue size exceeded, waiting for messages to be processed by paho client.
2025-01-06 22:42:55 - |WARNING| - [tb_device_mqtt.py] - tb_device_mqtt - __send_split_message - 925 - Queue size exceeded, waiting for messages to be processed by paho client.
2025-01-06 22:42:55 - |WARNING| - [tb_device_mqtt.py] - tb_device_mqtt - __send_split_message - 925 - Queue size exceeded, waiting for messages to be processed by paho client.
2025-01-06 22:42:55 - |WARNING| - [tb_device_mqtt.py] - tb_device_mqtt - __send_split_message - 925 - Queue size exceeded, waiting for messages to be processed by paho client.
2025-01-06 22:42:57 - |WARNING| - [slave.py] - slave - connect - 160 - Maximum attempt count (5) for device "IT50 - Pressa 26" - encountered.
2025-01-06 22:42:57 - |ERROR| - [modbus_connector.py] - modbus_connector - __poll_device - 239 - Socket is closed, connection is lost, for device IT50 - Pressa 26
2025-01-06 22:43:06 - |WARNING| - [tb_device_mqtt.py] - tb_device_mqtt - __send_split_message - 925 - Queue size exceeded, waiting for messages to be processed by paho client.
2025-01-06 22:43:06 - |WARNING| - [tb_device_mqtt.py] - tb_device_mqtt - __send_split_message - 925 - Queue size exceeded, waiting for messages to be processed by paho client.
2025-01-06 22:43:06 - |WARNING| - [tb_device_mqtt.py] - tb_device_mqtt - __send_split_message - 925 - Queue size exceeded, waiting for messages to be processed by paho client.
2025-01-06 22:43:17 - |WARNING| - [tb_device_mqtt.py] - tb_device_mqtt - __send_split_message - 925 - Queue size exceeded, waiting for messages to be processed by paho client.
2025-01-06 22:43:17 - |WARNING| - [tb_device_mqtt.py] - tb_device_mqtt - __send_split_message - 925 - Queue size exceeded, waiting for messages to be processed by paho client.
2025-01-06 22:43:17 - |WARNING| - [tb_device_mqtt.py] - tb_device_mqtt - __send_split_message - 925 - Queue size exceeded, waiting for messages to be processed by paho client.
2025-01-06 22:43:17 - |WARNING| - [tb_device_mqtt.py] - tb_device_mqtt - __send_split_message - 925 - Queue size exceeded, waiting for messages to be processed by paho client.
2025-01-06 22:43:28 - |WARNING| - [tb_device_mqtt.py] - tb_device_mqtt - __send_split_message - 925 - Queue size exceeded, waiting for messages to be processed by paho client.
2025-01-06 22:43:39 - |WARNING| - [tb_device_mqtt.py] - tb_device_mqtt - __send_split_message - 925 - Queue size exceeded, waiting for messages to be processed by paho client.
2025-01-06 22:43:39 - |WARNING| - [tb_device_mqtt.py] - tb_device_mqtt - __send_split_message - 925 - Queue size exceeded, waiting for messages to be processed by paho client.
2025-01-06 22:43:50 - |WARNING| - [tb_device_mqtt.py] - tb_device_mqtt - __send_split_message - 925 - Queue size exceeded, waiting for messages to be processed by paho client.
2025-01-06 22:43:59 - |WARNING| - [slave.py] - slave - connect - 160 - Maximum attempt count (5) for device "IT50 - Pressa 22" - encountered.
2025-01-06 22:43:59 - |ERROR| - [modbus_connector.py] - modbus_connector - __poll_device - 239 - Socket is closed, connection is lost, for device IT50 - Pressa 22
2025-01-06 22:44:01 - |WARNING| - [tb_device_mqtt.py] - tb_device_mqtt - __send_split_message - 925 - Queue size exceeded, waiting for messages to be processed by paho client.
2025-01-06 22:44:01 - |WARNING| - [tb_device_mqtt.py] - tb_device_mqtt - __send_split_message - 925 - Queue size exceeded, waiting for messages to be processed by paho client.
2025-01-06 22:44:01 - |WARNING| - [tb_device_mqtt.py] - tb_device_mqtt - __send_split_message - 925 - Queue size exceeded, waiting for messages to be processed by paho client.
2025-01-06 22:44:01 - |WARNING| - [tb_device_mqtt.py] - tb_device_mqtt - __send_split_message - 925 - Queue size exceeded, waiting for messages to be processed by paho client.
2025-01-06 22:44:12 - |WARNING| - [tb_device_mqtt.py] - tb_device_mqtt - __send_split_message - 925 - Queue size exceeded, waiting for messages to be processed by paho client.
2025-01-06 22:44:23 - |WARNING| - [tb_device_mqtt.py] - tb_device_mqtt - __send_split_message - 925 - Queue size exceeded, waiting for messages to be processed by paho client.
2025-01-06 22:44:23 - |WARNING| - [tb_device_mqtt.py] - tb_device_mqtt - __send_split_message - 925 - Queue size exceeded, waiting for messages to be processed by paho client.
2025-01-06 22:44:23 - |WARNING| - [tb_device_mqtt.py] - tb_device_mqtt - __send_split_message - 925 - Queue size exceeded, waiting for messages to be processed by paho client.
2025-01-06 22:44:23 - |WARNING| - [tb_device_mqtt.py] - tb_device_mqtt - __send_split_message - 925 - Queue size exceeded, waiting for messages to be processed by paho client.
2025-01-06 22:44:34 - |WARNING| - [tb_device_mqtt.py] - tb_device_mqtt - __send_split_message - 925 - Queue size exceeded, waiting for messages to be processed by paho client.
2025-01-06 22:44:34 - |WARNING| - [tb_device_mqtt.py] - tb_device_mqtt - __send_split_message - 925 - Queue size exceeded, waiting for messages to be processed by paho client.
2025-01-06 22:44:34 - |WARNING| - [tb_device_mqtt.py] - tb_device_mqtt - __send_split_message - 925 - Queue size exceeded, waiting for messages to be processed by paho client.
2025-01-06 22:44:34 - |WARNING| - [tb_device_mqtt.py] - tb_device_mqtt - __send_split_message - 925 - Queue size exceeded, waiting for messages to be processed by paho client.
2025-01-06 22:44:45 - |WARNING| - [tb_device_mqtt.py] - tb_device_mqtt - __send_split_message - 925 - Queue size exceeded, waiting for messages to be processed by paho client.
2025-01-06 22:44:45 - |WARNING| - [tb_device_mqtt.py] - tb_device_mqtt - __send_split_message - 925 - Queue size exceeded, waiting for messages to be processed by paho client.
2025-01-06 22:44:45 - |WARNING| - [tb_device_mqtt.py] - tb_device_mqtt - __send_split_message - 925 - Queue size exceeded, waiting for messages to be processed by paho client.
2025-01-06 22:44:45 - |WARNING| - [tb_device_mqtt.py] - tb_device_mqtt - __send_split_message - 925 - Queue size exceeded, waiting for messages to be processed by paho client.
2025-01-06 22:44:56 - |WARNING| - [tb_device_mqtt.py] - tb_device_mqtt - __send_split_message - 925 - Queue size exceeded, waiting for messages to be processed by paho client.
2025-01-06 22:44:56 - |WARNING| - [tb_device_mqtt.py] - tb_device_mqtt - __send_split_message - 925 - Queue size exceeded, waiting for messages to be processed by paho client.
2025-01-06 22:44:56 - |WARNING| - [tb_device_mqtt.py] - tb_device_mqtt - __send_split_message - 925 - Queue size exceeded, waiting for messages to be processed by paho client.
2025-01-06 22:44:56 - |WARNING| - [tb_device_mqtt.py] - tb_device_mqtt - __send_split_message - 925 - Queue size exceeded, waiting for messages to be processed by paho client.
2025-01-06 22:45:03 - |INFO| - [slave.py] - slave - connect - 166 - Connected to IT50 - Pressa 20
2025-01-06 22:45:03 - |ERROR| - [modbus_connector.py] - modbus_connector - __poll_device - 239 - Socket is closed, connection is lost, for device IT50 - Pressa 20
2025-01-06 22:45:07 - |WARNING| - [tb_device_mqtt.py] - tb_device_mqtt - __send_split_message - 925 - Queue size exceeded, waiting for messages to be processed by paho client.
2025-01-06 22:45:07 - |WARNING| - [tb_device_mqtt.py] - tb_device_mqtt - __send_split_message - 925 - Queue size exceeded, waiting for messages to be processed by paho client.
2025-01-06 22:45:07 - |WARNING| - [tb_device_mqtt.py] - tb_device_mqtt - __send_split_message - 925 - Queue size exceeded, waiting for messages to be processed by paho client.
2025-01-06 22:45:07 - |WARNING| - [tb_device_mqtt.py] - tb_device_mqtt - __send_split_message - 925 - Queue size exceeded, waiting for messages to be processed by paho client.
2025-01-06 22:45:18 - |WARNING| - [tb_device_mqtt.py] - tb_device_mqtt - __send_split_message - 925 - Queue size exceeded, waiting for messages to be processed by paho client.
2025-01-06 22:45:29 - |WARNING| - [tb_device_mqtt.py] - tb_device_mqtt - __send_split_message - 925 - Queue size exceeded, waiting for messages to be processed by paho client.
2025-01-06 22:45:29 - |WARNING| - [tb_device_mqtt.py] - tb_device_mqtt - __send_split_message - 925 - Queue size exceeded, waiting for messages to be processed by paho client.
2025-01-06 22:45:29 - |WARNING| - [tb_device_mqtt.py] - tb_device_mqtt - __send_split_message - 925 - Queue size exceeded, waiting for messages to be processed by paho client.
2025-01-06 22:45:29 - |WARNING| - [tb_device_mqtt.py] - tb_device_mqtt - __send_split_message - 925 - Queue size exceeded, waiting for messages to be processed by paho client.
2025-01-06 22:45:40 - |WARNING| - [tb_device_mqtt.py] - tb_device_mqtt - __send_split_message - 925 - Queue size exceeded, waiting for messages to be processed by paho client.
2025-01-06 22:45:51 - |WARNING| - [tb_device_mqtt.py] - tb_device_mqtt - __send_split_message - 925 - Queue size exceeded, waiting for messages to be processed by paho client.
2025-01-06 22:46:02 - |WARNING| - [tb_device_mqtt.py] - tb_device_mqtt - __send_split_message - 925 - Queue size exceeded, waiting for messages to be processed by paho client.
2025-01-06 22:46:02 - |WARNING| - [tb_device_mqtt.py] - tb_device_mqtt - __send_split_message - 925 - Queue size exceeded, waiting for messages to be processed by paho client.
2025-01-06 22:46:02 - |WARNING| - [tb_device_mqtt.py] - tb_device_mqtt - __send_split_message - 925 - Queue size exceeded, waiting for messages to be processed by paho client.
2025-01-06 22:46:13 - |WARNING| - [tb_device_mqtt.py] - tb_device_mqtt - __send_split_message - 925 - Queue size exceeded, waiting for messages to be processed by paho client.
2025-01-06 22:46:13 - |WARNING| - [tb_device_mqtt.py] - tb_device_mqtt - __send_split_message - 925 - Queue size exceeded, waiting for messages to be processed by paho client.
2025-01-06 22:46:13 - |WARNING| - [tb_device_mqtt.py] - tb_device_mqtt - __send_split_message - 925 - Queue size exceeded, waiting for messages to be processed by paho client.

Versions:

• OS: Docker
• Thingsboard IoT Gateway version 3.6.2

[imbeacon]

Hi @franklupo,

Thank you for your interest in ThingsBoard IoT Gateway.

Please provide information about how many datapoints you try to read per time?

In general this issue relates to Paho mqtt client, that we are using as a base for our Python MQTT client. It means that mqtt client send messages to the platform, but they were not acknowledged and it didn’t remove them. We discovered behaviour like this in version 3.5, but it wasn’t appear in our tests after that version. So please provide, if possible, your configuration to give us ability to reproduce this situation.

[franklupo]

do you mean modbus configuration?

Best regards

[imbeacon]

Yes, you can replace host and port if they are public.

[franklupo]

it60.json
it50.json

Best reagrds

[imbeacon]

Thank you for configuration, we will investigate this point, fix will be added in the one of next releases.
At the moment you can try to split devices to more connectors, like 2 devices per connector. But I’m not completely sure that it will help.