Right now, IO layers in aeronet only assume one thing: you can push bytes in, and pull bytes out of them; unordered and unreliable. This has turned out to actually be a pretty good design! You can easily add a new IO layer and you basically don't have to touch the stack higher up at all - things just keep working because aeronet_transport doesn't assume anything else other than that, and builds its own reliability and ordering on top (and importantly, a message ID and acknowledgement system).
However, different IO layers have different guarantees. UDP is always unreliable-unordered; WebSocket is always reliable-ordered; WebTransport may be unreliable-unordered or reliable-ordered depending on what channel type you use; you might have only one, or 1-or-more channels/lanes/streams available to use. The transport layer doesn't have to reimplement reliability and ordering on top of an already reliable and ordered channel, so it's a waste of bandwidth and unneeded overhead. What can we do?
Lanes and guarantees
Network IO code has generally converged on the concept of what I'm calling lanes (this might be called channels or streams in other implementations). A lane is a single bidirectional queue of messages between two endpoints which has its own reliability and ordering guarantees. We already have an implementation of lanes in aeronet_transport, but this is always a separate layer built on top of a generic underlying IO - it will always use a single unreliable-unordered physical lane on the underlying IO layer. I want to see if we can introduce this concept into the IO abstraction layer as well, and expose an IO implementation's capabilities to the transport layer, so it can make appropriate use of its physical lanes.
From the current IO implementations, it seems like there's a few categories of implementations:
| Category |
Lane multiplexing |
Reliable |
Ordered |
| 1 |
- |
- |
- |
| 2 |
Yes |
Yes |
Yes |
| 3 |
- |
Always |
Always |
Category 1 is raw UDP, UART (I know we don't have implementations for them, but we explicitly want to support this use-case). These IO layers have literally no guarantees apart from "push bytes in and they miiight come out on the other side", so we have to do everything ourselves. aeronet_transport currently assumes every IO layer acts like this. Actually, even worse: we don't get a guarantee that the data that gets sent is the same data that gets received byte-for-byte (we might get corruption, truncation, etc.).
Category 2 is QUIC, WebTransport + Iroh (which are built on QUIC), and Steam. These have a native idea of lanes, but we have to decide how many lanes we have, and what guarantees they get. aeronet_transport's model of lanes is largely based on QUIC streams, so that's a good candidate to implement for. However, the implementation of lanes varies wildly per IO layer - QUIC lets you open as many as you want, but with Steam you're limited to 255, and not recommended to go above 8: source here
At the time of this writing, some code has performance cost that is linear in the number of lanes, so keep the number of lanes to an absolute minimum. 3 or so is fine; >8 is a lot. The max number of lanes on Steam is 255, which is a very large number and not recommended!
Also interestingly:
Messages sent on a lane index other than 0 have a small overhead on the wire, so for maximum wire efficiency, lane 0 should be the "most common" lane, regardless of priorities or weights.
So honestly, it might not even be worth it to implement Steam-native lanes, and we should prefer aeronet_transport lanes over native lanes sometimes.
Category 3 is TCP and WebSocket. These have a single lane, but this lane is always reliable and ordered. This is currently the area where aeronet_io is the weakest, and where we waste a lot of bandwidth and overhead on re-implementing reliability and ordering on top of an already reliable-ordered channel.
So it seems like if we cover these three categories cleanly, we've solved ~99% of the issue and we don't have to care about an IO layer which is e.g. reliable but unordered.
Message IDs
Another thing to consider is that aeronet_transport introduces the concept of message IDs - you push a byte string, get a message ID, then you get an acknowledgement when that message ID has been received by the peer. If we use native lanes, that becomes harder to implement in a guarantee-agnostic way, and we're still left with some bandwidth overhead for tracking messages and IDs on both endpoints. And messages are an API-level guarantee, so either we keep this message ID tracking overhead, or we remove the concept of message IDs.
Right now, IO layers in aeronet only assume one thing: you can push bytes in, and pull bytes out of them; unordered and unreliable. This has turned out to actually be a pretty good design! You can easily add a new IO layer and you basically don't have to touch the stack higher up at all - things just keep working because
aeronet_transportdoesn't assume anything else other than that, and builds its own reliability and ordering on top (and importantly, a message ID and acknowledgement system).However, different IO layers have different guarantees. UDP is always unreliable-unordered; WebSocket is always reliable-ordered; WebTransport may be unreliable-unordered or reliable-ordered depending on what channel type you use; you might have only one, or 1-or-more channels/lanes/streams available to use. The transport layer doesn't have to reimplement reliability and ordering on top of an already reliable and ordered channel, so it's a waste of bandwidth and unneeded overhead. What can we do?
Lanes and guarantees
Network IO code has generally converged on the concept of what I'm calling lanes (this might be called channels or streams in other implementations). A lane is a single bidirectional queue of messages between two endpoints which has its own reliability and ordering guarantees. We already have an implementation of lanes in
aeronet_transport, but this is always a separate layer built on top of a generic underlying IO - it will always use a single unreliable-unordered physical lane on the underlying IO layer. I want to see if we can introduce this concept into the IO abstraction layer as well, and expose an IO implementation's capabilities to the transport layer, so it can make appropriate use of its physical lanes.From the current IO implementations, it seems like there's a few categories of implementations:
Category 1 is raw UDP, UART (I know we don't have implementations for them, but we explicitly want to support this use-case). These IO layers have literally no guarantees apart from "push bytes in and they miiight come out on the other side", so we have to do everything ourselves.
aeronet_transportcurrently assumes every IO layer acts like this. Actually, even worse: we don't get a guarantee that the data that gets sent is the same data that gets received byte-for-byte (we might get corruption, truncation, etc.).Category 2 is QUIC, WebTransport + Iroh (which are built on QUIC), and Steam. These have a native idea of lanes, but we have to decide how many lanes we have, and what guarantees they get.
aeronet_transport's model of lanes is largely based on QUIC streams, so that's a good candidate to implement for. However, the implementation of lanes varies wildly per IO layer - QUIC lets you open as many as you want, but with Steam you're limited to 255, and not recommended to go above 8: source hereAlso interestingly:
So honestly, it might not even be worth it to implement Steam-native lanes, and we should prefer
aeronet_transportlanes over native lanes sometimes.Category 3 is TCP and WebSocket. These have a single lane, but this lane is always reliable and ordered. This is currently the area where
aeronet_iois the weakest, and where we waste a lot of bandwidth and overhead on re-implementing reliability and ordering on top of an already reliable-ordered channel.So it seems like if we cover these three categories cleanly, we've solved ~99% of the issue and we don't have to care about an IO layer which is e.g. reliable but unordered.
Message IDs
Another thing to consider is that
aeronet_transportintroduces the concept of message IDs - you push a byte string, get a message ID, then you get an acknowledgement when that message ID has been received by the peer. If we use native lanes, that becomes harder to implement in a guarantee-agnostic way, and we're still left with some bandwidth overhead for tracking messages and IDs on both endpoints. And messages are an API-level guarantee, so either we keep this message ID tracking overhead, or we remove the concept of message IDs.