Skip to content

fix(runtime): keep messages between two processes in send order - #27

Merged
mremond merged 1 commit into
mainfrom
fm/task-84dc3f9e
Nov 13, 2026
Merged

mremond merged 1 commit into
mainfrom
fm/task-84dc3f9e

Conversation

@mremond

@mremond mremond commented Nov 13, 2026 •

Copy link
Copy Markdown
Member

Why

Erlang delivers the messages one process sends another in the order they were sent, but Nera sometimes swapped two of them, which made the receive_nil_message test fail now and then on macOS arm64.

What changed

  • A message sent to a process is never received before the messages the same sender sent it earlier.
  • The receive_nil_message test no longer fails intermittently.
  • A new end-to-end fixture checks message order between processes, natively and on BEAM.

Before and after

Before: On a macOS arm64 machine, receive_nil_message failed 14 of 300 runs with {badmatch,[[],[],e]}, the error seen in CI, and a probe streaming numbered messages saw reordering in 50 of 50 runs.

After: With the fix, receive_nil_message passed 500 of 500 runs, the probe saw no reordered message in 50 runs, and the new fixture passed 50 of 50.

Nothing on screen changes: this fixes the order in which the runtime delivers messages.

Checked

  • New unit tests that replay the interleaving, on the code before the fix: Both fail: the second message is received first ([2, 1]).
  • Same unit tests with the fix: Both pass.
  • send_order_between_processes fixture: Fails 5 of 5 runs before the fix, with out-of-order counts up to 378; passes 50 of 50 with the fix and passes on BEAM (OTP 29).
  • receive_nil_message on BEAM (OTP 29): Passes.
  • The project's checks on Linux x86_64 and macOS arm64: All passed: Rust checks, all three end-to-end parts on both systems, Cowlib, Thoas, Poolboy, macOS integration and cross-compiles.
How it was checked

Cause: a sender may copy a message straight into a ready or waiting receiver's mailbox. That path did not check the receiver's inbox, where an earlier message from the same sender could still wait after landing there while the receiver was running. The later message was then received first.

Fix: the direct path now requires an empty inbox, checked under the receiver's lock; otherwise the message joins the inbox behind the earlier ones.

cargo test --release -p nera-runtime --lib delivery_order, before and after the fix.

Loops of the compiled receive_nil_message test and of a probe streaming 4 x 200,000 numbered messages, before the fix (14 of 300 failed; 50 of 50 reordered) and after it (500 of 500 passed; 0 of 50 reordered). The after loops ran on the fix before the branch was brought up to date with main, whose two new commits change term copying and error/3; the project's checks ran at this exact version.

Risk

Risk above low: Blast radius, medium — Changes the send path every message between processes takes

Full risk assessment
Dimension Level Why
Blast radius medium Changes the send path every message between processes takes
Behavior low Corrects a delivery-order defect; Erlang semantics unchanged
Undo low Revert and done; no saved data involved
Evidence low Before and after loops, unit tests, fixture and green checks on both systems
Sensitivity low No sign-ins, keys, data or releases
Novelty low One extra condition on an existing fast path
Dependencies low No dependency changes

A sender may append a message straight to the receiver's mailbox when the
receiver is ready or waiting and its lock is free. That fast path did not
check the receiver's inbox. When the sender's earlier message had gone to
the inbox while the receiver was running, and the receiver then released
its lock before draining it, the later message landed in the mailbox first
and was received first.

receive_nil_message failed this way now and then on macOS arm64, receiving
[[], [], e] instead of [[], e, []].

The direct path now requires an empty inbox, checked under the receiver's
lock; otherwise the message goes to the inbox behind the earlier ones.
Unit tests replay the interleaving for plain and tuple sends, and the
send_order_between_processes fixture streams numbered messages between
several pairs of processes and checks their order natively and on BEAM.
@github-actions github-actions Bot added this to the 0.1.1 milestone Nov 13, 2026
@mremond
mremond merged commit 9c6da14 into main Nov 13, 2026
22 checks passed
@mremond
mremond deleted the fm/task-84dc3f9e branch November 14, 2026 11:03
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant