Repository navigation
fix(runtime): keep messages between two processes in send order - #27
Merged
Merged
Conversation
A sender may append a message straight to the receiver's mailbox when the receiver is ready or waiting and its lock is free. That fast path did not check the receiver's inbox. When the sender's earlier message had gone to the inbox while the receiver was running, and the receiver then released its lock before draining it, the later message landed in the mailbox first and was received first. receive_nil_message failed this way now and then on macOS arm64, receiving [[], [], e] instead of [[], e, []]. The direct path now requires an empty inbox, checked under the receiver's lock; otherwise the message goes to the inbox behind the earlier ones. Unit tests replay the interleaving for plain and tuple sends, and the send_order_between_processes fixture streams numbered messages between several pairs of processes and checks their order natively and on BEAM.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
Erlang delivers the messages one process sends another in the order they were sent, but Nera sometimes swapped two of them, which made the receive_nil_message test fail now and then on macOS arm64.
What changed
Before and after
Before: On a macOS arm64 machine, receive_nil_message failed 14 of 300 runs with {badmatch,[[],[],e]}, the error seen in CI, and a probe streaming numbered messages saw reordering in 50 of 50 runs.
After: With the fix, receive_nil_message passed 500 of 500 runs, the probe saw no reordered message in 50 runs, and the new fixture passed 50 of 50.
Nothing on screen changes: this fixes the order in which the runtime delivers messages.
Checked
How it was checked
Cause: a sender may copy a message straight into a ready or waiting receiver's mailbox. That path did not check the receiver's inbox, where an earlier message from the same sender could still wait after landing there while the receiver was running. The later message was then received first.
Fix: the direct path now requires an empty inbox, checked under the receiver's lock; otherwise the message joins the inbox behind the earlier ones.
cargo test --release -p nera-runtime --lib delivery_order, before and after the fix.
Loops of the compiled receive_nil_message test and of a probe streaming 4 x 200,000 numbered messages, before the fix (14 of 300 failed; 50 of 50 reordered) and after it (500 of 500 passed; 0 of 50 reordered). The after loops ran on the fix before the branch was brought up to date with main, whose two new commits change term copying and error/3; the project's checks ran at this exact version.
Risk
Risk above low: Blast radius, medium — Changes the send path every message between processes takes
Full risk assessment