diff --git a/active/0023-assets/Example.UML b/active/0023-assets/Example.UML new file mode 100644 index 0000000..c8cdf57 --- /dev/null +++ b/active/0023-assets/Example.UML @@ -0,0 +1,85 @@ +title QUIC Multi-Streams Setup Flow +actor Client +boundary Quicer +control ConnOwner +control EmqxConnection +control DataStream1 +control DataStream2 +autonumber 1 +== Init == +note over ConnOwner: Accepting New Connection +Client -> Quicer : QUIC: New connection +note over Quicer: Pick new connection owner +Quicer -> ConnOwner : {quic, new_conn, ... } +Activate ConnOwner +Client -> Quicer : QUIC: New Stream 1, with MQTT.Connect +Client -> Quicer : QUIC: New Stream 2, with MQTT.Subscribe +Client -> Quicer : QUIC: New Stream 3, with MQTT.Publish +create EmqxConnection +ConnOwner -> EmqxConnection: spawn +Deactivate ConnOwner +note right: Accepting Control Stream +EmqxConnection -> ConnOwner: initialized +Activate ConnOwner +ConnOwner -\\o Quicer: NIF call\n handshake +Deactivate ConnOwner +Quicer <----> Client: Complete Handshake +== QUIC Handshake Done == +Quicer -> EmqxConnection: {quic, new_stream, Stream1, ... } +note over EmqxConnection: Stream1 is Control Stream +activate EmqxConnection +Quicer -> EmqxConnection: {quic, Stream1, << MQTT.Connect ...>>} +Quicer -> ConnOwner: {quic, new_stream, Stream2, ... } +Activate ConnOwner +note over ConnOwner: Stream2 is DataStream +create DataStream1 +ConnOwner ----> DataStream1: Spawn +Quicer -> ConnOwner: {quic, new_stream, Stream3, ... } +note over ConnOwner: Stream3 is DataStream +ConnOwner <---> DataStream1: Ownership handoff +Deactivate DataStream1 +Note over DataStream1: Owner of Stream2\nrecv mode: passive \n Processing Pending +create DataStream2 +ConnOwner ----> DataStream2: Spawn +ConnOwner <---> DataStream2: Ownership handoff +Deactivate DataStream2 +Deactivate ConnOwner + +Note over DataStream2: Owner of Stream3\nrecv mode: passive\n Processing Pending + +note over EmqxConnection: Finish AUTH, Create Channel +EmqxConnection -\\o Quicer: NIF Send MQTT.ConnAck via Stream1 +Quicer -> Client: MQTT.ConnAck +== Control Stream Setup Success == +EmqxConnection -> ConnOwner: Call activate_data_streams +Activate ConnOwner +Note over ConnOwner: Activate pending data streams +ConnOwner -> DataStream1: activate with a copy of ChannelInfo +DataStream1 --> DataStream1: activate +ConnOwner -> DataStream2: activate with a copy of ChannelInfo +DataStream2 --> DataStream2: activate +ConnOwner -> EmqxConnection: reply call ok +Note over DataStream1: NIF call:\nsetopt active true +Note over DataStream2: NIF call:\nsetopt active true +deactivate ConnOwner +deactivate EmqxConnection +== Data Streams Activated == +{DataSub} Quicer -> DataStream1: {quic, Stream2, << MQTT.SUBSCRIBE... >> +Activate DataStream1 +Note over DataStream1: Processing MQTT.Subscribe +DataStream1 -\\o Quicer: NIF send: MQTT.SUBACK via Stream2 +Deactivate DataStream1 +Quicer -> Client: MQTT.SUBACK +{DataPub} Quicer -> DataStream2: {quic, Stream3, << MQTT.PUBLISH... >> +Activate DataStream2 +{DataSub} <-> {DataPub}: Can be in different ORDER +Note over DataStream2: Processing MQTT.Publish +DataStream2 -\\o Quicer: NIF send: MQTT.PUBACK via Stream3 +Quicer -> Client: MQTT.PUBACK +DataStream2 -> DataStream1: Publish Msg +Deactivate DataStream2 +DataStream1 -\\o Quicer: NIF call\nMQTT.Publish via Stream2 +Deactivate DataStream1 +Quicer -> Client: MQTT.Publish over Stream2 +Client -> Quicer: MQTT.PubAck over Stream2 +Quicer -> DataStream1: {quic, Stream2, << MQTT.PUBACK... >> diff --git a/active/0023-assets/Example.png b/active/0023-assets/Example.png new file mode 100644 index 0000000..e0a1b13 Binary files /dev/null and b/active/0023-assets/Example.png differ diff --git a/active/0023-quic-multi-streams.md b/active/0023-quic-multi-streams.md new file mode 100644 index 0000000..979f6f3 --- /dev/null +++ b/active/0023-quic-multi-streams.md @@ -0,0 +1,114 @@ +# MQTT over QUIC multstreams support + +## Changelog + +* 2022-12-13: @qzhuyan Initial draft + + +## Abstract + +MQTT Over QUIC in EMQX 5.0 is an experimental feature with single stream support. + +This EIP extends MQTT over QUIC to support multi-streams by utilizing the +multiplexing feature of QUIC transport protocol. + +## Motivation + +In EMQX5.0, Client and Server maintain a QUIC connection in between and +exchange MQTT messages over one single QUIC stream. When the stream +is closed by either side the connection is closed. + +By enabling multi-streams, one could achieve: + +1. Decouple connection control and application data exchanges. +1. Avoid head-of-line blocking among the Topics. +1. Decouple control plane traffic and data plane traffic. +1. Split channels for uplink data (publishing) and downlink data (receiving publish). +1. Priorities data from different MQTT Topics. +1. Improve parallelism of processing on the Client/Server side. +1. More robust handling of MQTT data that single stream abortion caused by the application will not cause connection close. +1. Flow control per data stream. +1. Reduce latency at application layer (application above MQTT) + A client does not need to wait for connack before sending the subscribe or publish packets. + +## Design + +The QUIC connection handshake between the MQTT client and EMQX is the same as the original single stream mode. + +Changes to support multi-streams. + +1. Bidirectional Control Stream, initiated from the MQTT Client. + There will be one and only one control stream from Client to Server. + + The first stream in the connection is the control stream. + + It has the same life cycle as the connection, either side of the connection closes the control + the stream leads to a connection close. + + Keep alive with MQTT.PING and authentication with MQTT.AUTH is over the control stream. + + All types of MQTT packets are allowed over the control stream which is backward compatible to the + original single stream mode. + +1. Bidirectional Data Streams, initiated from the MQTT Client. + + MQTT Client has the freedom to choose how to utilize the multiple data streams. + + Data stream could carry egress publishing traffic, ingress subscribed traffic, or both. + The data stream could carry data for one topic or from/to different topics. + + The client could also send the same topic data over different data streams but the receiving ordering is not guaranteed. + (Data ordering is only guaranteed in the same stream by the design of QUIC protocol) + + Zero or more data streams are allowed. The max number of data streams is decided by the broker (EMQX). + However, the client can request more allowed streams over QUIC protocol and it is upto the broker + to decide if increase the limit or not. + + Duplicated subscriptions are allowed among different streams. + + Data Stream only accepts a subset of MQTT packets and they are: + + - PUBLISH + - PUBACK + - PUBREC + - PUBCOMP + - PUBREL + - SUBSCRIBE + - SUBACK + - UNSUBSCRIBE + - UNSUBACK + +![](0023-assets/Example.png) + +## Configuration Changes + +At this stage, no. + +## Backwards Compatibility + +No Backwards Compatibility changes. Single stream mode is still supported. + +## Document Changes + +N/A + +## Testing Suggestions + +Multistream supports use more processes, the memory overhead need to be tested. + + +## Declined Alternatives + +1. In EMQX, use a single process to manage both the control stream and data streams + + This is the original idea used in PoC but it turns out to be too complicated to + manages the states per stream and requires major changes in `emqx_channel` and `emqx_routes` + + Using process per stream improves the parallelism, less blocking, and keeping + stream data and states isolated in each process is more secure and robust. + The overhead of memory footprints can be mitigated by using process hibernations. + + +1. Split sending/receiving in two unidirectional streams. + + Not approved at this stage.