Unable to upgrade RabbitMQ from 3.12.10 to 4.0.5 (required feature flag impossible to enable)

[Zoddo]

Community Support Policy

• I have read RabbitMQ's Community Support Policy
• I run RabbitMQ 4.x, the only series currently covered by community support
• I promise to provide all relevant information (versions, logs from all nodes, rabbitmq-diagnostics output, detailed reproduction steps)

RabbitMQ version used

4.0.5

Erlang version used

27.2.x

Operating system (distribution) used

ArchLinux

How is RabbitMQ deployed?

Other

rabbitmq-diagnostics status output

(status before upgrade, on 3.12.10)

Status of node rabbit@arch ...
Runtime

OS PID: 49229
OS: Linux
Uptime (seconds): 13
Is under maintenance?: false
RabbitMQ version: 3.12.10
RabbitMQ release series support status: supported
Node name: rabbit@arch
Erlang configuration: Erlang/OTP 27 [erts-15.2.1] [source] [64-bit] [smp:4:4] [ds:4:4:10] [async-threads:1] [jit:ns]
Crypto library: OpenSSL 3.4.0 22 Oct 2024
Erlang processes: 383 used, 1048576 limit
Scheduler run queue: 1
Cluster heartbeat timeout (net_ticktime): 60

Plugins

Enabled plugin file: /etc/rabbitmq/enabled_plugins
Enabled plugins:

 * rabbitmq_federation_management
 * rabbitmq_federation
 * rabbitmq_management
 * rabbitmq_management_agent
 * rabbitmq_web_dispatch
 * amqp_client
 * cowboy
 * cowlib

Data directory

Node data directory: /var/lib/rabbitmq/mnesia/rabbit@arch
Raft data directory: /var/lib/rabbitmq/mnesia/rabbit@arch/quorum/rabbit@arch

Config files


Log file(s)

 * /var/log/rabbitmq/[email protected]
 * <stdout>

Alarms

(none)

Memory

Total memory used: 0.1631 gb
Calculation strategy: rss
Memory high watermark setting: 0.4 of available memory, computed to: 3.1841 gb

reserved_unallocated: 0.0895 gb (54.88 %)
code: 0.0373 gb (22.89 %)
other_proc: 0.0235 gb (14.44 %)
other_system: 0.0208 gb (12.75 %)
other_ets: 0.003 gb (1.85 %)
atom: 0.0015 gb (0.93 %)
metrics: 0.0007 gb (0.45 %)
binary: 0.0005 gb (0.29 %)
plugins: 0.0003 gb (0.18 %)
msg_index: 0.0002 gb (0.1 %)
mgmt_db: 0.0001 gb (0.07 %)
mnesia: 0.0001 gb (0.05 %)
quorum_ets: 0.0 gb (0.02 %)
queue_procs: 0.0 gb (0.01 %)
connection_other: 0.0 gb (0.0 %)
quorum_queue_procs: 0.0 gb (0.0 %)
quorum_queue_dlx_procs: 0.0 gb (0.0 %)
stream_queue_procs: 0.0 gb (0.0 %)
stream_queue_replica_reader_procs: 0.0 gb (0.0 %)
connection_readers: 0.0 gb (0.0 %)
connection_writers: 0.0 gb (0.0 %)
connection_channels: 0.0 gb (0.0 %)
queue_slave_procs: 0.0 gb (0.0 %)
stream_queue_coordinator_procs: 0.0 gb (0.0 %)
allocated_unused: 0.0 gb (0.0 %)

File Descriptors

Total: 2, limit: 927
Sockets: 0, limit: 832

Free Disk Space

Low free disk space watermark: 0.05 gb
Free disk space: 157.6602 gb

Totals

Connection count: 0
Queue count: 1
Virtual host count: 1

Listeners

Interface: [::], port: 15672, protocol: http, purpose: HTTP API
Interface: [::], port: 25672, protocol: clustering, purpose: inter-node and CLI tool communication
Interface: 127.0.0.1, port: 5672, protocol: amqp, purpose: AMQP 0-9-1 and AMQP 1.0

Logs from node 1 (with sensitive values edited out)

févr. 06 22:24:25 arch rabbitmq[46205]: =INFO REPORT==== 6-Feb-2025::22:24:25.024776 ===
févr. 06 22:24:25 arch rabbitmq[46205]:     alarm_handler: {set,{system_memory_high_watermark,[]}}
févr. 06 22:24:28 arch rabbitmq[46205]: 2025-02-06 22:24:28.088408+01:00 [error] <0.218.0> Feature flags: `stream_filtering`: required feature flag not enabled! It must be enabled before upgrading RabbitMQ.
févr. 06 22:24:28 arch rabbitmq[46205]: 2025-02-06 22:24:28.097188+01:00 [error] <0.218.0> Failed to initialize feature flags registry: {disabled_required_feature_flag,
févr. 06 22:24:28 arch rabbitmq[46205]: 2025-02-06 22:24:28.097188+01:00 [error] <0.218.0>                                               stream_filtering}
févr. 06 22:24:28 arch rabbitmq[46205]: BOOT FAILED
févr. 06 22:24:28 arch rabbitmq[46205]: ===========
févr. 06 22:24:28 arch rabbitmq[46205]: Error during startup: {error,failed_to_initialize_feature_flags_registry}
févr. 06 22:24:28 arch rabbitmq[46205]: 2025-02-06 22:24:28.111559+01:00 [error] <0.218.0>
févr. 06 22:24:28 arch rabbitmq[46205]: 2025-02-06 22:24:28.111559+01:00 [error] <0.218.0> BOOT FAILED
févr. 06 22:24:28 arch rabbitmq[46205]: 2025-02-06 22:24:28.111559+01:00 [error] <0.218.0> ===========
févr. 06 22:24:28 arch rabbitmq[46205]: 2025-02-06 22:24:28.111559+01:00 [error] <0.218.0> Error during startup: {error,failed_to_initialize_feature_flags_registry}
févr. 06 22:24:28 arch rabbitmq[46205]: 2025-02-06 22:24:28.111559+01:00 [error] <0.218.0>
févr. 06 22:24:29 arch rabbitmq[46205]: 2025-02-06 22:24:29.113371+01:00 [notice] <0.45.0> Application rabbit exited with reason: {failed_to_initialize_feature_flags_registry,{rabbit,start,[normal,[]]}}
févr. 06 22:24:29 arch rabbitmq[46205]: {exit,terminating,[{application_controller,call,2,[{file,"application_controller.erl"},{line,511}]},{application,'-ensure_all_started/3-lc$^0/1-0-',1,[{file,"application.erl"},{line,367}]},{application,ensure_all_started,3,[{file,"application.erl"},{line,367}]},{rabbit,'-start_it/1-fun-0-',1,[{file,"rabbit.erl"},{line,430}]},{timer,tc,2,[{file,"timer.erl"},{line,595}]},{rabbit,start_it,1,[{file,"rabbit.erl"},{line,426}]},{init,start_it,1,[]},{init,start_em,1,[]}]}
févr. 06 22:24:29 arch rabbitmq[46205]: 2025-02-06 22:24:29.113811+01:00 [error] <0.217.0>   crasher:
févr. 06 22:24:29 arch rabbitmq[46205]: 2025-02-06 22:24:29.113811+01:00 [error] <0.217.0>     initial call: application_master:init/3
févr. 06 22:24:29 arch rabbitmq[46205]: 2025-02-06 22:24:29.113811+01:00 [error] <0.217.0>     pid: <0.217.0>
févr. 06 22:24:29 arch rabbitmq[46205]: 2025-02-06 22:24:29.113811+01:00 [error] <0.217.0>     registered_name: []
févr. 06 22:24:29 arch rabbitmq[46205]: 2025-02-06 22:24:29.113811+01:00 [error] <0.217.0>     exception exit: {failed_to_initialize_feature_flags_registry,
févr. 06 22:24:29 arch rabbitmq[46205]: 2025-02-06 22:24:29.113811+01:00 [error] <0.217.0>                         {rabbit,start,[normal,[]]}}
févr. 06 22:24:29 arch rabbitmq[46205]: 2025-02-06 22:24:29.113811+01:00 [error] <0.217.0>       in function  application_master:init/3 (application_master.erl, line 143)
févr. 06 22:24:29 arch rabbitmq[46205]: 2025-02-06 22:24:29.113811+01:00 [error] <0.217.0>     ancestors: [application_controller,<0.10.0>]
févr. 06 22:24:29 arch rabbitmq[46205]: 2025-02-06 22:24:29.113811+01:00 [error] <0.217.0>     message_queue_len: 1
févr. 06 22:24:29 arch rabbitmq[46205]: 2025-02-06 22:24:29.113811+01:00 [error] <0.217.0>     messages: [{'EXIT',<0.218.0>,normal}]
févr. 06 22:24:29 arch rabbitmq[46205]: 2025-02-06 22:24:29.113811+01:00 [error] <0.217.0>     links: [<0.45.0>]
févr. 06 22:24:29 arch rabbitmq[46205]: 2025-02-06 22:24:29.113811+01:00 [error] <0.217.0>     dictionary: []
févr. 06 22:24:29 arch rabbitmq[46205]: 2025-02-06 22:24:29.113811+01:00 [error] <0.217.0>     trap_exit: true
févr. 06 22:24:29 arch rabbitmq[46205]: 2025-02-06 22:24:29.113811+01:00 [error] <0.217.0>     status: running
févr. 06 22:24:29 arch rabbitmq[46205]: 2025-02-06 22:24:29.113811+01:00 [error] <0.217.0>     heap_size: 233
févr. 06 22:24:29 arch rabbitmq[46205]: 2025-02-06 22:24:29.113811+01:00 [error] <0.217.0>     stack_size: 29
févr. 06 22:24:29 arch rabbitmq[46205]: 2025-02-06 22:24:29.113811+01:00 [error] <0.217.0>     reductions: 67
févr. 06 22:24:29 arch rabbitmq[46205]: 2025-02-06 22:24:29.113811+01:00 [error] <0.217.0>   neighbours:
févr. 06 22:24:29 arch rabbitmq[46205]: 2025-02-06 22:24:29.113811+01:00 [error] <0.217.0>
févr. 06 22:24:29 arch rabbitmq[46205]: 2025-02-06 22:24:29.205241+01:00 [notice] <0.84.0>     alarm_handler: {clear,system_memory_high_watermark}
févr. 06 22:24:30 arch rabbitmq[46205]: Kernel pid terminated (application_controller) ("{application_start_failure,rabbit,{failed_to_initialize_feature_flags_registry,{rabbit,start,[normal,[]]}}}")
févr. 06 22:24:30 arch rabbitmq[46205]: 
févr. 06 22:24:30 arch rabbitmq[46205]: Crash dump is being written to: erl_crash.dump...done

Logs from node 2 (if applicable, with sensitive values edited out)

No response

Logs from node 3 (if applicable, with sensitive values edited out)

No response

rabbitmq.conf

# /etc/rabbitmq/rabbitmq-env.conf
NODENAME=rabbit@arch
NODE_IP_ADDRESS=127.0.0.1
NODE_PORT=5672

# /etc/rabbitmq/enabled_plugins
[rabbitmq_federation,rabbitmq_federation_management,rabbitmq_management].

Steps to deploy RabbitMQ cluster

RabbitMQ was installed from ArchLinux repos using pacman (pacman -S rabbitmq)

Steps to reproduce the behavior in question

Run RabbitMQ 3.12.10 (first installation was on an older version, I don't remember which one exactly), try to upgrade to 4.0.5.

advanced.config

No response

Application code

No response

Kubernetes deployment file

No response

What problem are you trying to solve?

After upgrading to 4.0.5, RabbitMQ crashed with Feature flags: `stream_filtering`: required feature flag not enabled! It must be enabled before upgrading RabbitMQ.

So I downgraded the package back to the previous version available (3.12.10) and ran:

# sudo rabbitmqctl enable_feature_flag all
Enabling all feature flags ...

I then upgraded again to 4.0.5... and was welcomed with the same error message.

So I downgraded again to 3.12.10 and checked feature flags:

# sudo rabbitmqctl list_feature_flags
Listing feature flags ...
name    state
classic_mirrored_queue_version  enabled
classic_queue_type_delivery_support     enabled
direct_exchange_routing_v2      enabled
drop_unroutable_metric  enabled
empty_basic_get_metric  enabled
feature_flags_v2        enabled
implicit_default_bindings       enabled
listener_records_in_ets enabled
maintenance_mode_status enabled
quorum_queue    enabled
restart_streams enabled
stream_queue    enabled
stream_sac_coordinator_unblock_group    enabled
stream_single_active_consumer   enabled
tracking_records_in_ets enabled
user_limits     enabled
virtual_host_metadata   enabled

As you can see, stream_filtering is completely missing. When I try to enable it explicitly, I get the following error message:

# sudo rabbitmqctl enable_feature_flag stream_filtering
Enabling feature flag "stream_filtering" ...
Error:
This feature flag is not supported by node rabbit@arch

So I'm not sure what's the correct way to upgrade rabbitmqctl. According to other issues/discussions, enabling all feature flags should be sufficient, but RabbitMQ 4.0 asks for a flag that, for some reason, doesn't exist on my 3.12 installation.

This is a dev instance, so I can probably just nuke the data directory and start over, but I'd like to understand the issue in case we have the same issue when we will deploy RabbitMQ on prod.

[SimonUnge]

Hi.

You need to upgrade 3.12 to 3.13, before upgrading to 4.0:

https://www.rabbitmq.com/docs/upgrade#rabbitmq-version-upgradability

[Zoddo]

Oh, I completely missed that 3.13 existed... because Arch repos have jumped directly from 3.12.10 to 4.0.5

Ok, so I think I will just end up emptying /var/lib/rabbitmq/ to start fresh, since running 3.13 to enable feature flags is going to be a pain (there is no existing package for this version that I can use).

Thanks!

[michaelklishin]

If you can afford to wipe a node, that's certainly the simplest option. For other cases there is Blue/Green deployment that can be done between two nodes on the same host if necessary.