[Questions] My queue does not receive messages when using the default exchange. After deleting and recreating the queue, it returns to normal.

[sheshesheshe]

Community Support Policy

• I have read RabbitMQ's Community Support Policy
• I run RabbitMQ 4.x, the only series currently covered by community support
• I promise to provide all relevant information (versions, logs from all nodes, rabbitmq-diagnostics output, detailed reproduction steps)

RabbitMQ version used

other (please specify)

Erlang version used

26.0.x

Operating system (distribution) used

k8s pod，rabbitmq: 3.12.14-management

How is RabbitMQ deployed?

Community Docker image

rabbitmq-diagnostics status output

Status of node rabbit@openstack-rabbitmq-rabbitmq-0.rabbitmqdiscovery.openstack.svc.cluster.local ...
Runtime

OS PID: 12
OS: Linux
Uptime (seconds): 300277
Is under maintenance?: false
RabbitMQ version: 3.12.14
RabbitMQ release series support status: out of support
Node name: rabbit@openstack-rabbitmq-rabbitmq-0.rabbitmqdiscovery.openstack.svc.cluster.local
Erlang configuration: Erlang/OTP 25 [erts-13.2.2.10] [source] [64-bit] [smp:1:1] [ds:1:1:10] [async-threads:1] [jit:ns]
Crypto library: OpenSSL 3.1.6 4 Jun 2024
Erlang processes: 6190 used, 1048576 limit
Scheduler run queue: 1
Cluster heartbeat timeout (net_ticktime): 60

Plugins

Enabled plugin file: /etc/rabbitmq/enabled_plugins
Enabled plugins:

 * rabbitmq_peer_discovery_k8s
 * rabbitmq_delayed_message_exchange
 * rabbitmq_peer_discovery_common
 * rabbitmq_tracing
 * rabbitmq_management
 * rabbitmq_management_agent
 * rabbitmq_web_dispatch
 * amqp_client
 * cowboy
 * cowlib

Data directory

Node data directory: /var/lib/rabbitmq/mnesia/rabbit@openstack-rabbitmq-rabbitmq-0.rabbitmqdiscovery.openstack.svc.cluster.local
Raft data directory: /var/lib/rabbitmq/mnesia/rabbit@openstack-rabbitmq-rabbitmq-0.rabbitmqdiscovery.openstack.svc.cluster.local/quorum/rabbit@openstack-rabbitmq-rabbitmq-0.rabbitmqdiscovery.openstack.svc.cluster.local

Config files

 * /etc/rabbitmq/rabbitmq.conf
 * /etc/rabbitmq/conf.d/10-defaults.conf

Log file(s)

 * <stdout>

Alarms

(none)

Memory

Total memory used: 1.2117 gb
Calculation strategy: rss
Memory high watermark setting: 0.4 of available memory, computed to: 107.9315 gb

binary: 0.365 gb (30.12 %)
reserved_unallocated: 0.1912 gb (15.78 %)
quorum_ets: 0.1539 gb (12.7 %)
allocated_unused: 0.123 gb (10.15 %)
other_proc: 0.1034 gb (8.53 %)
quorum_queue_procs: 0.0795 gb (6.56 %)
plugins: 0.0429 gb (3.54 %)
code: 0.0358 gb (2.95 %)
connection_other: 0.0284 gb (2.35 %)
other_ets: 0.0268 gb (2.21 %)
mgmt_db: 0.0213 gb (1.76 %)
other_system: 0.0173 gb (1.43 %)
connection_readers: 0.0114 gb (0.94 %)
connection_channels: 0.0039 gb (0.33 %)
mnesia: 0.0027 gb (0.23 %)
metrics: 0.0022 gb (0.18 %)
atom: 0.0019 gb (0.15 %)
connection_writers: 0.0007 gb (0.06 %)
msg_index: 0.0004 gb (0.03 %)
quorum_queue_dlx_procs: 0.0 gb (0.0 %)
stream_queue_procs: 0.0 gb (0.0 %)
stream_queue_replica_reader_procs: 0.0 gb (0.0 %)
queue_procs: 0.0 gb (0.0 %)
queue_slave_procs: 0.0 gb (0.0 %)
stream_queue_coordinator_procs: 0.0 gb (0.0 %)

File Descriptors

Total: 2097, limit: 1048479
Sockets: 310, limit: 943629

Free Disk Space

Low free disk space watermark: 0.05 gb
Free disk space: 862.5229 gb

Totals

Connection count: 310
Queue count: 640
Virtual host count: 13

Listeners

Interface: [::], port: 15680, protocol: https, purpose: HTTP API over TLS (HTTPS)
Interface: [::], port: 25672, protocol: clustering, purpose: inter-node and CLI tool communication
Interface: [::], port: 5672, protocol: amqp/ssl, purpose: AMQP 0-9-1 and AMQP 1.0 over TLS

Logs from node 1 (with sensitive values edited out)

2025-03-14 15:40:52.186856+00:00 [info] <0.17335.0> queue 'reply_worker10759:nova-compute:1' in vhost 'nova': granting vote for {'nova_reply_worker10759:nova-compute:1','rabbit@openstack-rabbitmq-rabbitmq-2.rabbitmqdiscovery.openstack.svc.cluster.local'} with last indexterm {0,0} for term 1 previous term was 0
2025-03-14 15:40:52.226899+00:00 [info] <0.17335.0> queue 'reply_worker10759:nova-compute:1' in vhost 'nova': detected a new leader {'nova_reply_worker10759:nova-compute:1','rabbit@openstack-rabbitmq-rabbitmq-2.rabbitmqdiscovery.openstack.svc.cluster.local'} in term 1
2025-03-17 07:16:57.299338+00:00 [notice] <0.17335.0> queue 'reply_worker10759:nova-compute:1' in vhost 'nova': terminating with reason 'delete'
2025-03-17 07:17:02.303891+00:00 [info] <0.20822.1508> Deleting server 'nova_reply_worker10759:nova-compute:1' and its data directory.
2025-03-17 07:37:15.102620+00:00 [info] <0.3295.1516> queue 'reply_worker10759:nova-compute:1' in vhost 'nova': granting vote for {'nova_reply_worker10759:nova-compute:1','rabbit@openstack-rabbitmq-rabbitmq-2.rabbitmqdiscovery.openstack.svc.cluster.local'} with last indexterm {0,0} for term 1 previous term was 0
2025-03-17 07:37:15.123754+00:00 [info] <0.3295.1516> queue 'reply_worker10759:nova-compute:1' in vhost 'nova': detected a new leader {'nova_reply_worker10759:nova-compute:1','rabbit@openstack-rabbitmq-rabbitmq-2.rabbitmqdiscovery.openstack.svc.cluster.local'} in term 1
2025-03-17 07:37:34.523393+00:00 [info] <0.3295.1516> queue 'reply_worker10759:nova-compute:1' in vhost 'nova': granting vote for {'nova_reply_worker10759:nova-compute:1','rabbit@openstack-rabbitmq-rabbitmq-1.rabbitmqdiscovery.openstack.svc.cluster.local'} with last indexterm {602735,1} for term 2 previous term was 1
2025-03-17 07:37:34.555033+00:00 [info] <0.3295.1516> queue 'reply_worker10759:nova-compute:1' in vhost 'nova': follower did not have entry at 602735 in 1. Requesting {'nova_reply_worker10759:nova-compute:1','rabbit@openstack-rabbitmq-rabbitmq-1.rabbitmqdiscovery.openstack.svc.cluster.local'} from 62
2025-03-17 07:37:34.555186+00:00 [info] <0.3295.1516> queue 'reply_worker10759:nova-compute:1' in vhost 'nova': detected a new leader {'nova_reply_worker10759:nova-compute:1','rabbit@openstack-rabbitmq-rabbitmq-1.rabbitmqdiscovery.openstack.svc.cluster.local'} in term 2

Logs from node 2 (if applicable, with sensitive values edited out)

2025-03-17 07:37:34.523151+00:00 [notice] <0.2161.0> queue 'reply_worker10759:nova-compute:1' in vhost 'nova': candidate -> leader in term: 2 machine version: 3

Logs from node 3 (if applicable, with sensitive values edited out)

2025-03-14 15:40:52.172887+00:00 [notice] <0.16346.0> queue 'reply_worker10759:nova-compute:1' in vhost 'nova': candidate -> leader in term: 1 machine version: 3
2025-03-14 15:40:52.173245+00:00 [info] <0.16332.0> ra: started cluster nova_reply_worker10759:nova-compute:1 with 3 servers
2025-03-14 15:40:52.173245+00:00 [info] <0.16332.0> Leader: {'nova_reply_worker10759:nova-compute:1','rabbit@openstack-rabbitmq-rabbitmq-2.rabbitmqdiscovery.openstack.svc.cluster.local'}
2025-03-17 07:16:57.298114+00:00 [notice] <0.16346.0> queue 'reply_worker10759:nova-compute:1' in vhost 'nova': leader -> terminating_leader in term: 1 machine version: 3
2025-03-17 07:17:02.299979+00:00 [info] <0.3296.1489> Deleting server 'nova_reply_worker10759:nova-compute:1' and its data directory.
2025-03-17 07:17:02.302695+00:00 [warning] <0.3296.1489> Force delete of {'nova_reply_worker10759:nova-compute:1','rabbit@openstack-rabbitmq-rabbitmq-1.rabbitmqdiscovery.openstack.svc.cluster.local'} failed with: {error,{badrpc,nodedown}}This may require manual data clean up
2025-03-17 07:17:02.303891+00:00 [info] <16297.20822.1508> Deleting server 'nova_reply_worker10759:nova-compute:1' and its data directory.
2025-03-17 07:37:15.072638+00:00 [error] <0.13957.1495> ra: failed to start a server #{cluster_name => 'nova_reply_worker10759:nova-compute:1',friendly_name => [113,117,101,117,101,32,39,114,101,112,108,121,95,119,111,114,107,101,114,49,48,55,53,57,58,110,111,118,97,45,99,111,109,112,117,116,101,58,49,39,32,105,110,32,118,104,111,115,116,32,39,110,111,118,97,39],id => {'nova_reply_worker10759:nova-compute:1','rabbit@openstack-rabbitmq-rabbitmq-1.rabbitmqdiscovery.openstack.svc.cluster.local'},initial_members => [{'nova_reply_worker10759:nova-compute:1','rabbit@openstack-rabbitmq-rabbitmq-2.rabbitmqdiscovery.openstack.svc.cluster.local'},{'nova_reply_worker10759:nova-compute:1','rabbit@openstack-rabbitmq-rabbitmq-1.rabbitmqdiscovery.openstack.svc.cluster.local'},{'nova_reply_worker10759:nova-compute:1','rabbit@openstack-rabbitmq-rabbitmq-0.rabbitmqdiscovery.openstack.svc.cluster.local'}],log_init_args => #{snapshot_interval => 8192,uid => <<78,79,86,65,95,82,54,57,78,73,50,89,70,49,90,81,67,53>>},machine => {module,rabbit_fifo,#{become_leader_handler => {rabbit_quorum_queue,become_leader,[{resource,<<110,111,118,97>>,queue,<<114,101,112,108,121,95,119,111,114,107,101,114,49,48,55,53,57,58,110,111,118,97,45,99,111,109,112,117,116,101,58,49>>}]},created => 1742197035071,dead_letter_handler => undefined,delivery_limit => undefined,expires => 1800000,max_bytes => undefined,max_in_memory_bytes => undefined,max_in_memory_length => undefined,max_length => undefined,msg_ttl => undefined,name => 'nova_reply_worker10759:nova-compute:1',overflow_strategy => drop_head,queue_resource => {resource,<<110,111,118,97>>,queue,<<114,101,112,108,121,95,119,111,114,107,101,114,49,48,55,53,57,58,110,111,118,97,45,99,111,109,112,117,116,101,58,49>>},single_active_consumer_on => false}},metrics_key => {resource,<<110,111,118,97>>,queue,<<114,101,112,108,121,95,119,111,114,107,101,114,49,48,55,53,57,58,110,111,118,97,45,99,111,109,112,117,116,101,58,49>>},ra_event_formatter => {rabbit_quorum_queue,format_ra_event,[{resource,<<110,111,118,97>>,queue,<<114,101,112,108,121,95,119,111,114,107,101,114,49,48,55,53,57,58,110,111,118,97,45,99,111,109,112,117,116,101,58,49>>}]},tick_timeout => 5000,uid => <<78,79,86,65,95,82,54,57,78,73,50,89,70,49,90,81,67,53>>}, error: {error,
2025-03-17 07:37:15.123444+00:00 [notice] <0.13956.1495> queue 'reply_worker10759:nova-compute:1' in vhost 'nova': candidate -> leader in term: 1 machine version: 3
2025-03-17 07:37:15.123744+00:00 [info] <0.13945.1495> ra: started cluster nova_reply_worker10759:nova-compute:1 with 3 servers
2025-03-17 07:37:15.123744+00:00 [info] <0.13945.1495> 1 servers failed to start: [{'nova_reply_worker10759:nova-compute:1','rabbit@openstack-rabbitmq-rabbitmq-1.rabbitmqdiscovery.openstack.svc.cluster.local'}]
2025-03-17 07:37:15.123744+00:00 [info] <0.13945.1495> Leader: {'nova_reply_worker10759:nova-compute:1','rabbit@openstack-rabbitmq-rabbitmq-2.rabbitmqdiscovery.openstack.svc.cluster.local'}
2025-03-17 07:37:34.519260+00:00 [notice] <0.13956.1495> queue 'reply_worker10759:nova-compute:1' in vhost 'nova': leader saw append_entries_reply from {'nova_reply_worker10759:nova-compute:1','rabbit@openstack-rabbitmq-rabbitmq-1.rabbitmqdiscovery.openstack.svc.cluster.local'} for term 2 abdicates term: 1!
2025-03-17 07:37:34.520739+00:00 [notice] <0.13956.1495> queue 'reply_worker10759:nova-compute:1' in vhost 'nova': leader -> follower in term: 2 machine version: 3
2025-03-17 07:37:34.520928+00:00 [info] <0.13956.1495> queue 'reply_worker10759:nova-compute:1' in vhost 'nova': granting vote for {'nova_reply_worker10759:nova-compute:1','rabbit@openstack-rabbitmq-rabbitmq-1.rabbitmqdiscovery.openstack.svc.cluster.local'} with last indexterm {602735,1} for term 2 previous term was 2
2025-03-17 07:37:34.523283+00:00 [info] <0.13956.1495> queue 'reply_worker10759:nova-compute:1' in vhost 'nova': follower did not have entry at 602735 in 1. Requesting {'nova_reply_worker10759:nova-compute:1','rabbit@openstack-rabbitmq-rabbitmq-1.rabbitmqdiscovery.openstack.svc.cluster.local'} from 62
2025-03-17 07:37:34.523424+00:00 [info] <0.13956.1495> queue 'reply_worker10759:nova-compute:1' in vhost 'nova': detected a new leader {'nova_reply_worker10759:nova-compute:1','rabbit@openstack-rabbitmq-rabbitmq-1.rabbitmqdiscovery.openstack.svc.cluster.local'} in term 2
2025-03-17 07:50:13.979384+00:00 [info] <0.13956.1495> queue 'reply_worker10759:nova-compute:1' in vhost 'nova': follower received leader command from <16297.5260.1494>. Rejecting to {'nova_reply_worker10759:nova-compute:1','rabbit@openstack-rabbitmq-rabbitmq-1.rabbitmqdiscovery.openstack.svc.cluster.local'}

rabbitmq.conf

 enabled_plugins: |2


    [rabbitmq_management,rabbitmq_peer_discovery_k8s,rabbitmq_delayed_message_exchange,rabbitmq_tracing].
  rabbitmq-env.conf: |
    SERVER_ADDITIONAL_ERL_ARGS="+S 1:1"
  rabbitmq.conf: |
    cluster_formation.k8s.address_type = hostname
    cluster_formation.k8s.host = kubernetes.default.svc.cluster.local
    cluster_formation.node_cleanup.interval = 10
    cluster_formation.node_cleanup.only_log_warning = true
    cluster_formation.peer_discovery_backend = rabbit_peer_discovery_k8s
    cluster_partition_handling = autoheal
    listeners.ssl.1 = 5672
    listeners.tcp = none
    log.console = true
    loopback_users.guest = false
    management.ssl.cacertfile = /etc/rabbitmq/certs/ca.crt
    management.ssl.certfile = /etc/rabbitmq/certs/tls.crt
    management.ssl.keyfile = /etc/rabbitmq/certs/tls.key
    management.load_definitions = /var/lib/rabbitmq/definitions.json
    management.ssl.port = 15680
    queue_master_locator = min-masters
    ssl_options.cacertfile = /etc/rabbitmq/certs/ca.crt
    ssl_options.certfile = /etc/rabbitmq/certs/tls.crt
    ssl_options.fail_if_no_peer_cert = true
    ssl_options.keyfile = /etc/rabbitmq/certs/tls.key
    ssl_options.verify = verify_peer

Steps to deploy RabbitMQ cluster

chart：https://opendev.org/openstack/openstack-helm-infra/src/branch/master/rabbitmq
helm upgrade openstack-rabbitmq openstack/rabbitmq --install --create-namespace --namespace openstack --version 3.12.14 --set pod.replicas.server=3 

Steps to reproduce the behavior in question

I have a 3 node RabbitMQ cluster. All queues are quorum type.
If one node disappears offline (crash/network fail/hard powerdown/etc) for about 30 minutes before coming back online.
I have the following situation：The message was sent correctly to the default exchange, but it did not reach the queue
The raw data I collected using tracing is as follows:

2025-03-18T00:59:04.689+00:00: Message published

Node:         rabbit@openstack-rabbitmq-rabbitmq-2.rabbitmqdiscovery.openstack.svc.cluster.local
Connection:   10.244.0.81:42552 -> 10.244.2.18:5672
Virtual host: nova
User:         nova
Channel:      1
Exchange:     
Routing keys: [<<"reply_worker10759:nova-compute:1">>]
Routed queues: [<<"reply_worker10759:nova-compute:1">>]
Properties:   [{<<"priority">>,signedint,0},
               {<<"delivery_mode">>,signedint,2},
               {<<"headers">>,table,[]},
               {<<"content_encoding">>,longstr,<<"utf-8">>},
               {<<"content_type">>,longstr,<<"application/json">>}]
Payload: 
{"oslo.version": "2.0", "oslo.message": "{\"result\": {\"nova_object.name\": \"ConfigHIK\", \"nova_object.namespace\": \"nova\", \"nova_object.version\": \"1.0\", \"nova_object.data\": {\"id\": 18, \"key\": \"pci_white_list\", \"value\": \"[{\\\"product_name\\\": \\\"Tesla P4\\\", \\\"product_id\\\": \\\"1bb3\\\", \\\"vendor_id\\\": \\\"10de\\\", \\\"device_func\\\": \\\"type-GPU\\\"}, {\\\"product_name\\\": \\\"Tesla P40\\\", \\\"product_id\\\": \\\"1b38\\\", \\\"vendor_id\\\": \\\"10de\\\", \\\"device_func\\\": \\\"type-GPU\\\"}, {\\\"product_name\\\": \\\"Tesla T4\\\", \\\"product_id\\\": \\\"1eb8\\\", \\\"vendor_id\\\": \\\"10de\\\", \\\"device_func\\\": \\\"type-GPU\\\"}, {\\\"product_name\\\": \\\"Device 0220\\\", \\\"product_id\\\": \\\"0220\\\", \\\"vendor_id\\\": \\\"cabc\\\", \\\"device_func\\\": \\\"type-GPU\\\"}, {\\\"product_name\\\": \\\"Device 3559\\\", \\\"product_id\\\": \\\"3559\\\", \\\"vendor_id\\\": \\\"19e5\\\", \\\"device_func\\\": \\\"type-GPU\\\"}, {\\\"product_name\\\": \\\"Intel 82599\\\", \\\"product_id\\\": \\\"10fb\\\", \\\"vendor_id\\\": \\\"8086\\\", \\\"device_func\\\": \\\"type-NIC\\\"}, {\\\"product_name\\\": \\\"Intel X722SPF+\\\", \\\"product_id\\\": \\\"37d0\\\", \\\"vendor_id\\\": \\\"8086\\\", \\\"device_func\\\": \\\"type-NIC\\\"}, {\\\"product_name\\\": \\\"Intel X722\\\", \\\"product_id\\\": \\\"37d1\\\", \\\"vendor_id\\\": \\\"8086\\\", \\\"device_func\\\": \\\"type-NIC\\\"}, {\\\"product_name\\\": \\\"Intel I350\\\", \\\"product_id\\\": \\\"1521\\\", \\\"vendor_id\\\": \\\"8086\\\", \\\"device_func\\\": \\\"type-NIC\\\"}, {\\\"product_name\\\": \\\"Intel X710\\\", \\\"product_id\\\": \\\"1572\\\", \\\"vendor_id\\\": \\\"8086\\\", \\\"device_func\\\": \\\"type-NIC\\\"}, {\\\"product_name\\\": \\\"Huawei HNS RDMA Network Controller\\\", \\\"product_id\\\": \\\"a222\\\", \\\"vendor_id\\\": \\\"19e5\\\", \\\"device_func\\\": \\\"type-NIC\\\"}, {\\\"product_name\\\": \\\"Huawei HNS Network Controller\\\", \\\"product_id\\\": \\\"a221\\\", \\\"vendor_id\\\": \\\"19e5\\\", \\\"device_func\\\": \\\"type-NIC\\\"}, {\\\"product_name\\\": \\\"Mellanox ConnectX-4\\\", \\\"product_id\\\": \\\"1015\\\", \\\"vendor_id\\\": \\\"15b3\\\", \\\"device_func\\\": \\\"type-NIC\\\"}]\", \"project_id\": \"admin\", \"run_state\": false, \"created_at\": null, \"updated_at\": null, \"deleted_at\": null, \"deleted\": false}}, \"failure\": null, \"ending\": true, \"_msg_id\": \"4ab2ce17d5214a4bad29293ce8813062\", \"_unique_id\": \"4dc61d563af542c0b2d47851f43b4629\"}"}

What problem are you trying to solve?

I have a 3 node RabbitMQ cluster. All queues are quorum type.
If one node disappears offline (crash/network fail/hard powerdown/etc) for about 30 minutes before coming back online.
I have the following situation：The message was sent correctly to the default exchange, but it did not reach the queue
The raw data I collected using tracing is as follows:

2025-03-18T00:59:04.689+00:00: Message published

Node:         rabbit@openstack-rabbitmq-rabbitmq-2.rabbitmqdiscovery.openstack.svc.cluster.local
Connection:   10.244.0.81:42552 -> 10.244.2.18:5672
Virtual host: nova
User:         nova
Channel:      1
Exchange:     
Routing keys: [<<"reply_worker10759:nova-compute:1">>]
Routed queues: [<<"reply_worker10759:nova-compute:1">>]
Properties:   [{<<"priority">>,signedint,0},
               {<<"delivery_mode">>,signedint,2},
               {<<"headers">>,table,[]},
               {<<"content_encoding">>,longstr,<<"utf-8">>},
               {<<"content_type">>,longstr,<<"application/json">>}]
Payload: 
{"oslo.version": "2.0", "oslo.message": "{\"result\": {\"nova_object.name\": \"ConfigHIK\", \"nova_object.namespace\": \"nova\", \"nova_object.version\": \"1.0\", \"nova_object.data\": {\"id\": 18, \"key\": \"pci_white_list\", \"value\": \"[{\\\"product_name\\\": \\\"Tesla P4\\\", \\\"product_id\\\": \\\"1bb3\\\", \\\"vendor_id\\\": \\\"10de\\\", \\\"device_func\\\": \\\"type-GPU\\\"}, {\\\"product_name\\\": \\\"Tesla P40\\\", \\\"product_id\\\": \\\"1b38\\\", \\\"vendor_id\\\": \\\"10de\\\", \\\"device_func\\\": \\\"type-GPU\\\"}, {\\\"product_name\\\": \\\"Tesla T4\\\", \\\"product_id\\\": \\\"1eb8\\\", \\\"vendor_id\\\": \\\"10de\\\", \\\"device_func\\\": \\\"type-GPU\\\"}, {\\\"product_name\\\": \\\"Device 0220\\\", \\\"product_id\\\": \\\"0220\\\", \\\"vendor_id\\\": \\\"cabc\\\", \\\"device_func\\\": \\\"type-GPU\\\"}, {\\\"product_name\\\": \\\"Device 3559\\\", \\\"product_id\\\": \\\"3559\\\", \\\"vendor_id\\\": \\\"19e5\\\", \\\"device_func\\\": \\\"type-GPU\\\"}, {\\\"product_name\\\": \\\"Intel 82599\\\", \\\"product_id\\\": \\\"10fb\\\", \\\"vendor_id\\\": \\\"8086\\\", \\\"device_func\\\": \\\"type-NIC\\\"}, {\\\"product_name\\\": \\\"Intel X722SPF+\\\", \\\"product_id\\\": \\\"37d0\\\", \\\"vendor_id\\\": \\\"8086\\\", \\\"device_func\\\": \\\"type-NIC\\\"}, {\\\"product_name\\\": \\\"Intel X722\\\", \\\"product_id\\\": \\\"37d1\\\", \\\"vendor_id\\\": \\\"8086\\\", \\\"device_func\\\": \\\"type-NIC\\\"}, {\\\"product_name\\\": \\\"Intel I350\\\", \\\"product_id\\\": \\\"1521\\\", \\\"vendor_id\\\": \\\"8086\\\", \\\"device_func\\\": \\\"type-NIC\\\"}, {\\\"product_name\\\": \\\"Intel X710\\\", \\\"product_id\\\": \\\"1572\\\", \\\"vendor_id\\\": \\\"8086\\\", \\\"device_func\\\": \\\"type-NIC\\\"}, {\\\"product_name\\\": \\\"Huawei HNS RDMA Network Controller\\\", \\\"product_id\\\": \\\"a222\\\", \\\"vendor_id\\\": \\\"19e5\\\", \\\"device_func\\\": \\\"type-NIC\\\"}, {\\\"product_name\\\": \\\"Huawei HNS Network Controller\\\", \\\"product_id\\\": \\\"a221\\\", \\\"vendor_id\\\": \\\"19e5\\\", \\\"device_func\\\": \\\"type-NIC\\\"}, {\\\"product_name\\\": \\\"Mellanox ConnectX-4\\\", \\\"product_id\\\": \\\"1015\\\", \\\"vendor_id\\\": \\\"15b3\\\", \\\"device_func\\\": \\\"type-NIC\\\"}]\", \"project_id\": \"admin\", \"run_state\": false, \"created_at\": null, \"updated_at\": null, \"deleted_at\": null, \"deleted\": false}}, \"failure\": null, \"ending\": true, \"_msg_id\": \"4ab2ce17d5214a4bad29293ce8813062\", \"_unique_id\": \"4dc61d563af542c0b2d47851f43b4629\"}"}

I can manually use the routing key "reply_worker10759:nova-compute:1" to retrieve the message in the corresponding "reply_worker10759:nova-compute:1" queue.

The queue "reply_worker10759:nova-compute:1" does not consistently fail to retrieve message; occasionally, it can retrieve message. I am not sure of the specific reason, but when I delete and recreate the queue in the RabbitMQ management, the environment returns to normal.
Below is the message obtained from the queue, and the traced data:

2025-03-18T01:01:32.893+00:00: Message published

Node:         rabbit@openstack-rabbitmq-rabbitmq-2.rabbitmqdiscovery.openstack.svc.cluster.local
Connection:   10.244.0.81:52026 -> 10.244.2.18:5672
Virtual host: nova
User:         nova
Channel:      1
Exchange:     
Routing keys: [<<"reply_worker10759:nova-compute:1">>]
Routed queues: [<<"reply_worker10759:nova-compute:1">>]
Properties:   [{<<"priority">>,signedint,0},
               {<<"delivery_mode">>,signedint,2},
               {<<"headers">>,table,[]},
               {<<"content_encoding">>,longstr,<<"utf-8">>},
               {<<"content_type">>,longstr,<<"application/json">>}]
Payload: 
{"oslo.version": "2.0", "oslo.message": "{\"result\": {\"nova_object.name\": \"ConfigHIK\", \"nova_object.namespace\": \"nova\", \"nova_object.version\": \"1.0\", \"nova_object.data\": {\"id\": 18, \"key\": \"pci_white_list\", \"value\": \"[{\\\"product_name\\\": \\\"Tesla P4\\\", \\\"product_id\\\": \\\"1bb3\\\", \\\"vendor_id\\\": \\\"10de\\\", \\\"device_func\\\": \\\"type-GPU\\\"}, {\\\"product_name\\\": \\\"Tesla P40\\\", \\\"product_id\\\": \\\"1b38\\\", \\\"vendor_id\\\": \\\"10de\\\", \\\"device_func\\\": \\\"type-GPU\\\"}, {\\\"product_name\\\": \\\"Tesla T4\\\", \\\"product_id\\\": \\\"1eb8\\\", \\\"vendor_id\\\": \\\"10de\\\", \\\"device_func\\\": \\\"type-GPU\\\"}, {\\\"product_name\\\": \\\"Device 0220\\\", \\\"product_id\\\": \\\"0220\\\", \\\"vendor_id\\\": \\\"cabc\\\", \\\"device_func\\\": \\\"type-GPU\\\"}, {\\\"product_name\\\": \\\"Device 3559\\\", \\\"product_id\\\": \\\"3559\\\", \\\"vendor_id\\\": \\\"19e5\\\", \\\"device_func\\\": \\\"type-GPU\\\"}, {\\\"product_name\\\": \\\"Intel 82599\\\", \\\"product_id\\\": \\\"10fb\\\", \\\"vendor_id\\\": \\\"8086\\\", \\\"device_func\\\": \\\"type-NIC\\\"}, {\\\"product_name\\\": \\\"Intel X722SPF+\\\", \\\"product_id\\\": \\\"37d0\\\", \\\"vendor_id\\\": \\\"8086\\\", \\\"device_func\\\": \\\"type-NIC\\\"}, {\\\"product_name\\\": \\\"Intel X722\\\", \\\"product_id\\\": \\\"37d1\\\", \\\"vendor_id\\\": \\\"8086\\\", \\\"device_func\\\": \\\"type-NIC\\\"}, {\\\"product_name\\\": \\\"Intel I350\\\", \\\"product_id\\\": \\\"1521\\\", \\\"vendor_id\\\": \\\"8086\\\", \\\"device_func\\\": \\\"type-NIC\\\"}, {\\\"product_name\\\": \\\"Intel X710\\\", \\\"product_id\\\": \\\"1572\\\", \\\"vendor_id\\\": \\\"8086\\\", \\\"device_func\\\": \\\"type-NIC\\\"}, {\\\"product_name\\\": \\\"Huawei HNS RDMA Network Controller\\\", \\\"product_id\\\": \\\"a222\\\", \\\"vendor_id\\\": \\\"19e5\\\", \\\"device_func\\\": \\\"type-NIC\\\"}, {\\\"product_name\\\": \\\"Huawei HNS Network Controller\\\", \\\"product_id\\\": \\\"a221\\\", \\\"vendor_id\\\": \\\"19e5\\\", \\\"device_func\\\": \\\"type-NIC\\\"}, {\\\"product_name\\\": \\\"Mellanox ConnectX-4\\\", \\\"product_id\\\": \\\"1015\\\", \\\"vendor_id\\\": \\\"15b3\\\", \\\"device_func\\\": \\\"type-NIC\\\"}]\", \"project_id\": \"admin\", \"run_state\": false, \"created_at\": null, \"updated_at\": null, \"deleted_at\": null, \"deleted\": false}}, \"failure\": null, \"ending\": true, \"_msg_id\": \"5f91c90c8394456a822fae739dbc58f6\", \"_unique_id\": \"bec22f1d9888435caf7967121f58b4c6\"}"}

2025-03-18T01:01:32.902+00:00: Message received

Node:         rabbit@openstack-rabbitmq-rabbitmq-2.rabbitmqdiscovery.openstack.svc.cluster.local
Connection:   172.37.125.74:20940 -> 10.244.2.18:5672
Virtual host: nova
User:         nova
Channel:      1
Exchange:     
Routing keys: [<<"reply_worker10759:nova-compute:1">>]
Queue:        reply_worker10759:nova-compute:1
Properties:   [{<<"priority">>,signedint,0},
               {<<"delivery_mode">>,signedint,2},
               {<<"headers">>,table,[]},
               {<<"content_encoding">>,longstr,<<"utf-8">>},
               {<<"content_type">>,longstr,<<"application/json">>}]
Payload: 
{"oslo.version": "2.0", "oslo.message": "{\"result\": {\"nova_object.name\": \"ConfigHIK\", \"nova_object.namespace\": \"nova\", \"nova_object.version\": \"1.0\", \"nova_object.data\": {\"id\": 18, \"key\": \"pci_white_list\", \"value\": \"[{\\\"product_name\\\": \\\"Tesla P4\\\", \\\"product_id\\\": \\\"1bb3\\\", \\\"vendor_id\\\": \\\"10de\\\", \\\"device_func\\\": \\\"type-GPU\\\"}, {\\\"product_name\\\": \\\"Tesla P40\\\", \\\"product_id\\\": \\\"1b38\\\", \\\"vendor_id\\\": \\\"10de\\\", \\\"device_func\\\": \\\"type-GPU\\\"}, {\\\"product_name\\\": \\\"Tesla T4\\\", \\\"product_id\\\": \\\"1eb8\\\", \\\"vendor_id\\\": \\\"10de\\\", \\\"device_func\\\": \\\"type-GPU\\\"}, {\\\"product_name\\\": \\\"Device 0220\\\", \\\"product_id\\\": \\\"0220\\\", \\\"vendor_id\\\": \\\"cabc\\\", \\\"device_func\\\": \\\"type-GPU\\\"}, {\\\"product_name\\\": \\\"Device 3559\\\", \\\"product_id\\\": \\\"3559\\\", \\\"vendor_id\\\": \\\"19e5\\\", \\\"device_func\\\": \\\"type-GPU\\\"}, {\\\"product_name\\\": \\\"Intel 82599\\\", \\\"product_id\\\": \\\"10fb\\\", \\\"vendor_id\\\": \\\"8086\\\", \\\"device_func\\\": \\\"type-NIC\\\"}, {\\\"product_name\\\": \\\"Intel X722SPF+\\\", \\\"product_id\\\": \\\"37d0\\\", \\\"vendor_id\\\": \\\"8086\\\", \\\"device_func\\\": \\\"type-NIC\\\"}, {\\\"product_name\\\": \\\"Intel X722\\\", \\\"product_id\\\": \\\"37d1\\\", \\\"vendor_id\\\": \\\"8086\\\", \\\"device_func\\\": \\\"type-NIC\\\"}, {\\\"product_name\\\": \\\"Intel I350\\\", \\\"product_id\\\": \\\"1521\\\", \\\"vendor_id\\\": \\\"8086\\\", \\\"device_func\\\": \\\"type-NIC\\\"}, {\\\"product_name\\\": \\\"Intel X710\\\", \\\"product_id\\\": \\\"1572\\\", \\\"vendor_id\\\": \\\"8086\\\", \\\"device_func\\\": \\\"type-NIC\\\"}, {\\\"product_name\\\": \\\"Huawei HNS RDMA Network Controller\\\", \\\"product_id\\\": \\\"a222\\\", \\\"vendor_id\\\": \\\"19e5\\\", \\\"device_func\\\": \\\"type-NIC\\\"}, {\\\"product_name\\\": \\\"Huawei HNS Network Controller\\\", \\\"product_id\\\": \\\"a221\\\", \\\"vendor_id\\\": \\\"19e5\\\", \\\"device_func\\\": \\\"type-NIC\\\"}, {\\\"product_name\\\": \\\"Mellanox ConnectX-4\\\", \\\"product_id\\\": \\\"1015\\\", \\\"vendor_id\\\": \\\"15b3\\\", \\\"device_func\\\": \\\"type-NIC\\\"}]\", \"project_id\": \"admin\", \"run_state\": false, \"created_at\": null, \"updated_at\": null, \"deleted_at\": null, \"deleted\": false}}, \"failure\": null, \"ending\": true, \"_msg_id\": \"5f91c90c8394456a822fae739dbc58f6\", \"_unique_id\": \"bec22f1d9888435caf7967121f58b4c6\"}"}

[michaelklishin]

RabbitMQ 3.12 is out of community support.

[michaelklishin]

@sheshesheshe according to the logs on node 3, one of the quorum queues was deleted. There is nothing else in the logs.

I have the following situation：The message was sent correctly to the default exchange,
but it did not reach the queue

The number of scenarios is broad. See Publisher-side Data Safety and Unroutable Message Handling.

If a queue with the target name does not exist at the time of publishing, the default exchange will route it nowhere, that is, at best the message can only be return to the publisher (if it opts in)
and no publisher confirm will be sent in response to such a publish.

That's as much time as I'm willing to spend on a 3.12.x question.