RIP‐80 Priority Message

Status

• Current State: Discussing

• Authors: imzs

• Mailing List discussion: [email protected]

Background & Motivation

Apache RocketMQ lacks a native message priority feature. Although this is not a widely used functionality, there are specific scenarios where such a feature is required. For example, in resource-constrained consumption scenarios, classification-based resource allocation is needed to ensure that important payloads are processed with higher priority. Currently, RocketMQ users achieve this by splitting topics and allocating different processing resources to different topics (e.g., assigning more threads to consume high-priority topics), which is not flexible or convenient.

What do we need to do

• Will we add a new module? No.

• Will we add new APIs? Yes.

• Will we add a new feature? Yes.

Why should we do that

To provide users with a more flexible and convenient way to handle message priority processing.

Goals

• Provide a native priority mechanism that's more user-friendly than current workarounds

• Enable producers to set priority for individual messages based on business requirements

• Allow pop consumers to process higher priority messages before lower priority ones

Non-Goals

• Supporting priority consumption for pull consumer

• Guaranteeing absolute priority ordering in all scenarios

• Global priority ordering across all brokers in the cluster

Changes

Interface Design/Change

A new Message APIs to support message priority settings, mark on system properties.

Implementation

1. A New Message Type: Introduce a new message type - priority messages, conceptually parallel to transactional messages and scheduled messages.

2. Priority Definition: Following JMS and AMQP standards, support numeric priority levels within a defined range. Support both ascending and descending priority ordering configurations.

3. Priority settings will be applied per message and stored in the message's system properties by the Producer.

4. Priority-to-Queue Mapping: Map priority numbers to queue IDs, where each priority level corresponds to a dedicated ConsumeQueue. This naturally separates messages of different priority levels. And priority level between 1 and 10 are highly recommended.

5. Consumption Strategy: Change the current random queue selection strategy to sequential scanning. Messages will be fetched preferentially from higher priority queues to achieve priority-based consumption.

6. Retry Policy: Policy configurable, retry messages have lower priority than normal messages by default.

7. Consumption Mode Limitation: The sequential queue scanning strategy requires that clients cannot bind to specific queues (rebalance by topic) or specify queue id to fetch. Therefore, this feature will be implemented based on pop consumption (including kv pop) and will not support pull mode priority consumption.

8. Distributed Considerations: Since RocketMQ is a distributed service with messages spread across multiple brokers, we only focus on priority handling within individual brokers. We do not plan to implement global sorting of messages across all brokers. While lower priority messages may occasionally be processed before existing higher priority messages under certain conditions, the macro view shows that most high priority messages are processed first, which satisfies basic user requirements. This approach has been validated in production environments.

Compatibility, Deprecation, and Migration Plan

• Are backward and forward compatibility taken into consideration?

No

• Are there deprecated APIs?

No

• How do we do migration?

No need

Rejected Alternatives

How do alternatives solve the issue you proposed?

Current workarounds include topic splitting and manual resource allocation.

Pros and Cons of alternatives

Why should we reject above alternatives

While topic splitting works, it's not a user-friendly solution and requires significant manual intervention. A native priority feature provides a more elegant and manageable approach.