Repository navigation
Say what a clean determinism sweep could not have seen - #265
Merged
bbondy merged 1 commit intoSep 13, 2026
Merged
Conversation
The weekly measurement reports that every test agreed with itself and says nothing about what that sentence can carry. Ten runs cannot account for a test that loses one run in ten, and thirty only just can, so a clean summary reads as an all-clear when all it ever was is a bound. That gap is not theoretical. A ten run sweep of the suite on a hosted runner named nothing at four threads and nothing at sixteen. The same script, at the same `--test-threads=4`, in a rust:slim container on a busy sixteen core host, named nine tests in crates/agent/tests/turn.rs, one of them failing four runs in ten. Same flag, same tree, different machine, and neither report carried what a reader needs to tell those two results apart. So every report now states the CPU count of the machine that produced it and the failure rate that many runs could have seen: above that rate a test was unlikely to get through unseen, below it the odds of a clean sweep are worse than one in twenty. An issue body carries the machine for the same reason, since the rate in it is a property of that machine as much as of the test. The comment claiming the wide leg is what `make check` runs as on the machines people run it on was wrong. Sixteen threads on a four core runner is those cores oversubscribed, not a stand-in for a sixteen core machine, and the mock server races `make check-linux` caps threads for want a machine with something else running on it, which a hosted runner does not have. An issue filed by hand no longer credits the workflow for it. Reading a rate under a job's name and finding it was somebody's laptop is a reader trusting an environment that never ran the test. The selftest is 22 checks, three of them new: that a clean report states its own bound, that a wider sweep claims a tighter one, and that only a run which was a job may claim the workflow filed it.
bbondy
approved these changes
Sep 13, 2026
bbondy
marked this pull request as ready for review
September 13, 2026 21:57
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The problem
The determinism workflow added in #261 reports
Every test agreed with itself across all N runsandgives a reader nothing to weigh that against. Ten runs cannot account for a test that loses one run
in ten: such a test passes ten in a row 35% of the time. Thirty, the count it shipped with, reaches
only about a tenth, which is where every flake seen in this suite sits. The summary's strongest
sentence is a bound presented as a result, and the rate it could have seen was nowhere in it.
Reproduce
gh workflow run test-determinism.yml -f runs=10contrib/measure-flakes.py --runs 10 --test-threads 4The runner named nothing, at four threads and again at sixteen. The second, in a
rust:slimcontainer on a busy sixteen core host, named nine tests in
crates/agent/tests/turn.rs, one of themfailing 4 of 10. The two reports are the same shape, and neither states the machine or the
sensitivity, so nothing in either explains why they disagree. Expected: a report that says what it
was able to see.
The fix
Each report now names the CPU count of the machine that produced it and the failure rate that many
runs could have detected: above that rate a test was unlikely to get through unseen, below it a clean
sweep is worse than a one in twenty coincidence. An issue body carries the machine for the same
reason, because the rate in it belongs to that machine as much as to the test. The comment claiming
the wide leg is what
make checkruns as on a developer's machine was wrong, and says instead thatsixteen threads on four cores is oversubscription. Raising the count itself is #267, so that the
sentence about what a sweep could see lands before the sweep it describes gets longer. An issue filed by hand no longer credits the
workflow for it.
What the container found, and why it wears one disguise
Six of the nine failed with
Connection refusedagainst the mock server.serveandserve_sequencein turn.rs bind a listener, move it into a thread that accepts a fixed number ofconnections, and let it drop when that thread returns, so any retry by the client meets a closed port
whatever the original cause was. The 4-of-10 case is
left: 204, right: 205: a turn one round short,which is the shape of a response cut off when the server drops the stream after writing it. Both are
test harness defects rather than slowness, and neither is fixed here.
Test plan
contrib/measure-flakes.py --selftest- 22 checks passed, 3 of them newmake check-reviewdog- no findingsmake check-spec- 55 specs, 518 clauses, 0 errors(1 - p) ** nruby -ryamlparse of the workflow after the comment change