Skip to content

Say what a clean determinism sweep could not have seen - #265

Merged
bbondy merged 1 commit into
brave:mainfrom
netzenbot:report-what-a-flake-sweep-could-not-have-seen
Sep 13, 2026
Merged

bbondy merged 1 commit into
brave:mainfrom
netzenbot:report-what-a-flake-sweep-could-not-have-seen

Conversation

@netzenbot

@netzenbot netzenbot commented Sep 13, 2026 •

Copy link
Copy Markdown
Collaborator

The problem

The determinism workflow added in #261 reports Every test agreed with itself across all N runs and
gives a reader nothing to weigh that against. Ten runs cannot account for a test that loses one run
in ten: such a test passes ten in a row 35% of the time. Thirty, the count it shipped with, reaches
only about a tenth, which is where every flake seen in this suite sits. The summary's strongest
sentence is a bound presented as a result, and the rate it could have seen was nowhere in it.

Reproduce

  1. Measure with a short count on a hosted runner: gh workflow run test-determinism.yml -f runs=10
  2. Measure at the same parallelism on a machine with other work on it:
    contrib/measure-flakes.py --runs 10 --test-threads 4

The runner named nothing, at four threads and again at sixteen. The second, in a rust:slim
container on a busy sixteen core host, named nine tests in crates/agent/tests/turn.rs, one of them
failing 4 of 10. The two reports are the same shape, and neither states the machine or the
sensitivity, so nothing in either explains why they disagree. Expected: a report that says what it
was able to see.

The fix

Each report now names the CPU count of the machine that produced it and the failure rate that many
runs could have detected: above that rate a test was unlikely to get through unseen, below it a clean
sweep is worse than a one in twenty coincidence. An issue body carries the machine for the same
reason, because the rate in it belongs to that machine as much as to the test. The comment claiming
the wide leg is what make check runs as on a developer's machine was wrong, and says instead that
sixteen threads on four cores is oversubscription. Raising the count itself is #267, so that the
sentence about what a sweep could see lands before the sweep it describes gets longer. An issue filed by hand no longer credits the
workflow for it.

What the container found, and why it wears one disguise

Six of the nine failed with Connection refused against the mock server. serve and
serve_sequence in turn.rs bind a listener, move it into a thread that accepts a fixed number of
connections, and let it drop when that thread returns, so any retry by the client meets a closed port
whatever the original cause was. The 4-of-10 case is left: 204, right: 205: a turn one round short,
which is the shape of a response cut off when the server drops the stream after writing it. Both are
test harness defects rather than slowness, and neither is fixed here.

Test plan

  • contrib/measure-flakes.py --selftest - 22 checks passed, 3 of them new
  • make check-reviewdog - no findings
  • make check-spec - 55 specs, 518 clauses, 0 errors
  • Rendered reports at several run counts, checking each stated bound against (1 - p) ** n
  • ruby -ryaml parse of the workflow after the comment change
  • No Rust changed, so the suite and MSRV were not re-run; the pre-commit hook ran fmt and clippy
  • CI passes cleanly

The weekly measurement reports that every test agreed with itself and
says nothing about what that sentence can carry. Ten runs cannot account
for a test that loses one run in ten, and thirty only just can, so a
clean summary reads as an all-clear when all it ever was is a bound.

That gap is not theoretical. A ten run sweep of the suite on a hosted
runner named nothing at four threads and nothing at sixteen. The same
script, at the same `--test-threads=4`, in a rust:slim container on a
busy sixteen core host, named nine tests in crates/agent/tests/turn.rs,
one of them failing four runs in ten. Same flag, same tree, different
machine, and neither report carried what a reader needs to tell those
two results apart.

So every report now states the CPU count of the machine that produced it
and the failure rate that many runs could have seen: above that rate a
test was unlikely to get through unseen, below it the odds of a clean
sweep are worse than one in twenty. An issue body carries the machine
for the same reason, since the rate in it is a property of that machine
as much as of the test.

The comment claiming the wide leg is what `make check` runs as on the
machines people run it on was wrong. Sixteen threads on a four core
runner is those cores oversubscribed, not a stand-in for a sixteen core
machine, and the mock server races `make check-linux` caps threads for
want a machine with something else running on it, which a hosted runner
does not have.

An issue filed by hand no longer credits the workflow for it. Reading a
rate under a job's name and finding it was somebody's laptop is a reader
trusting an environment that never ran the test.

The selftest is 22 checks, three of them new: that a clean report states
its own bound, that a wider sweep claims a tighter one, and that only a
run which was a job may claim the workflow filed it.
@bbondy
bbondy marked this pull request as ready for review September 13, 2026 21:57
@bbondy
bbondy merged commit 03dfef7 into brave:main Sep 13, 2026
18 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants