Skip to content

test(tcp): fail tcp_reactor_perf only on a recurring stall - #28

Merged
mremond merged 4 commits into
mainfrom
fm/task-0b5601e6
Nov 13, 2026
Merged

mremond merged 4 commits into
mainfrom
fm/task-0b5601e6

Conversation

@mremond

@mremond mremond commented Nov 13, 2026 •

Copy link
Copy Markdown
Member

Why

The macOS end-to-end check failed whenever one 200-round batch of the TCP re-arm loop ran five times slower than the median, and short bursts of host activity or brief whole-program pauses are enough to do that.

What changed

  • The TCP reactor performance test fails only when a stall or slowdown shows in at least two batches.
  • Batches are timed in microseconds, and the stall check alternates ten batches between two connections.
  • The project status page records the hosted measurements and why one slow batch is no longer a failure.

Before and after

Before: One slow batch failed the test: on hosted macOS runners the old checks failed about 1 run in 110, with no single round anywhere near the reactor's 100 ms fallback.

After: The new test passed every clean run on a loaded Mac and on this pull request's own build, and every injected recurring stall, first-round stall and slowdown still failed it.

This changes only how the project is checked; nothing a person sees changes.

Checked

Live validation: Demonstrated for this version. 6 of 6 scenarios driven live in the product.

Scenario Result Evidence
Old and new test alternated hundreds of times on a loaded Mac with hosted macOS builds ✅ Passed live Old test failed 1 of 1,200 runs, new test 0 of 1,200
Test variants with a 100 ms stall on each connection's first round, a recurring 100 ms stall, or 1 ms slowdowns from round 300 or 600 ✅ Passed live Every variant failed every run, on main's build and on this pull request's build
One 100 ms pause injected in the middle of a run ✅ Passed live New test passed every run; the old test failed every run
The committed test run ten times in a row with this pull request's own macOS build ✅ Passed live 10 of 10 passed with no retries
The project's hosted checks on this pull request ✅ Passed live Every end-to-end shard on macOS and Linux, the Rust checks and both compiler builds passed
Every round timed on three hosted macOS runners with main's build, beside a 1 ms timer that does not use the TCP reactor, with delay and pause controls ✅ Passed live Without extra load no round reached 90 ms in 1,800 runs, and every round of 20 ms or more matched a whole-program pause; under full processor load, isolated rounds sat at 126 to 137 ms, none near the reactor's 100 ms fallback
  • Unchanged test on a loaded Mac, and an equivalent loop on OTP 29 under the same load: Nera builds failed 0 to 3 of 300 runs each; OTP 29 failed 3 of 300
  • Per-round probe on three hosted macOS runners, 1,800 runs without extra load: No round reached 90 ms; old checks failed 16 runs from bursts and whole-program pauses of 21 to 53 ms
  • Same probe with every processor busy, 300 runs: 14 isolated rounds of 126 to 137 ms and one 133 ms whole-program pause; none near 100 ms
  • Full end-to-end suite with an earlier main build: 481 passed, 2 skipped for missing external modules
How it was checked

Hosted checks passed on this version, including every end-to-end shard on macOS and Linux.

An independent review found nothing blocking; its wording corrections are included.

Risk

Risk: low in every dimension.

Full risk assessment
Dimension Level Why
Blast radius low One end-to-end test file and a status paragraph
Behavior low Nothing changes for people using Nera; only the check's failure rule
Undo low Revert the commit
Evidence low Hosted checks pass; a hosted probe, repeated runs and injected stalls show the rule
Sensitivity low No sign-ins, keys, data or releases
Novelty low Same loop and ratios, regrouped batches
Dependencies low No dependencies added or updated
Not done

A real stall that happens only once per run does not fail the test; the hosted probe found no such stall without extra load.

A stall that hits only one of the three sustained batches no longer fails the throughput check.

The new rules' failure rate on hosted runners is not measured; estimates from the probe data suggest a few runs in a thousand.

The cause of the isolated 126 to 137 ms rounds under full processor load is not proven.

Pipeline

Updates from git push no-mistakes

Infinitary checked this version: an independent review that also judged the documents, and live validation the review accepted.

@github-actions github-actions Bot added this to the 0.1.1 milestone Nov 13, 2026
The batch-consistency check failed whenever one batch took more than
five times the median. A single host pause can do that at any time: on a
loaded macOS host the same loop on OTP 29 failed the rule too, and
unchanged Nera builds failed it in roughly one run in 250.

Both checks now time batches in microseconds and fail only when the
slowdown shows in at least two batches. The consistency check runs ten
batches alternating between two connections, so a stall on every
connection's first re-arm still fails it. The throughput check compares
each of three sustained batches with the warmup.
A per-round probe on hosted macOS runners found single slow batches coming from bursts that slow many rounds and from whole-program pauses of 20 to 55 ms, with no round near the reactor's 100 ms poll timeout. The test header and the status paragraph now say so, and the status paragraph records the unexplained 126 to 137 ms rounds seen under full CPU load.
The hosted probe's 16 failures in 1,800 runs counted either old check. The batch rule failed about one run in 225 and the old throughput check about one run in 180; the status paragraph now says so.
@mremond
mremond merged commit 4e7ad86 into main Nov 13, 2026
20 checks passed
@mremond
mremond deleted the fm/task-0b5601e6 branch November 14, 2026 11:02
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant