Repository navigation
Measure test determinism rather than guessing which tests flake - #261
Merged
Merged
Conversation
checks.md tells anybody hitting a failure that does not reproduce to name it and carry on, which is right, and nothing in this repo ever measured which tests actually behave that way. So the answer to "is that a flake" was a shrug, and it stayed a shrug however many times the same test lost a run. contrib/measure-flakes.py builds the suite once, runs that build N times, tallies every test's outcome per run, and reports each test whose outcome changed with the rate it changed at and the panic it produced. Cargo announces each test binary on stderr while libtest writes outcomes on stdout, so the two streams are read as one: without the announcement a name like `a_reply_arrives_before_the_deadline` cannot be told apart from the same name in another binary. Every test it names then gets an issue titled `Flaky test: <name>`, which is the title the one flake anybody here filed by hand already carries, so a test that already has an issue gets nothing further whoever opened it. A rate fails nothing: a test failing three runs in thirty is exactly what the policy above describes, and going red over it would teach people to ignore the job. The exit code is nonzero only when the measurement did not happen, or when an issue that was meant to be filed was not, since a filer that has quietly stopped working looks exactly like a suite that stopped being flaky. The new Test determinism workflow is the first scheduled one here. It runs weekly against the default branch, files what it finds, and leaves the rates in the job summary. A pull request touching the script runs its selftest and nothing else, since an hour of suite runs in front of a review is worthless and a parser that matches nothing would otherwise report a clean suite forever. It measures at four threads and at sixteen, both named rather than left to cargo. A hosted runner has four cores, so cargo's default there is already the cap `make check-linux` sets, and the wide leg is what `make check` runs as on the machines people run it on. The difference between the two legs is what says whether that cap belongs in more than one target. Only the wide leg files, since both legs measure the same tests and both filing would race to open two issues for one flake. Evidence on a 16 core macOS host: ten runs of all 3558 tests at default parallelism found nothing that disagreed with itself. A test written to lose every other run was reported at 50 percent with its panic, and what it would have filed was checked against the live issue list: the known flake matched brave#76 and was left alone, and only the new name would have been opened. Closes brave#223
netzenbot
force-pushed
the
fix-measure-test-determinism
branch
from
September 13, 2026 19:16
f19a97c to
229884a
Compare
bbondy
marked this pull request as ready for review
September 13, 2026 21:06
bbondy
approved these changes
Sep 13, 2026
6 of 7 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #223
The problem
Nobody can say which tests here are flaky.
docs/development/checks.mdtells anyone hitting afailure that does not reproduce to name it and carry on, which is good advice in front of a red
build and also a policy that can absorb a real regression: "it reproduces on the parent, carry on"
is only safe if somebody knows which tests behave that way. The one class anybody can point at,
Text file busy (os error 26), is known because two people happened to file it (#18, #76).Reproduce
grep -rn "cargo test" .github/workflows/docs/development/checks.md, then try to answer "which tests, andhow often?"
Every job runs the suite once, so there is nothing to read: a flake stays a shrug however many runs
it costs. Expected: a list of names with rates.
With this branch,
contrib/measure-flakes.py --runs 10reported3558 tests reported. Every test agreed with itself across all 10 runs.on a 16 core macOS host. With a test written to lose everyother run it reported
flake_demo::a_quiet_test_loses_a_run_now_and_then | 2 | 4 | 50%and itspanic, and kept a test that failed every run out of that table. What it would have filed was checked
against the live issue list: the known flake matched #76 and was left alone.
contrib/measure-flakes.py --selftestpins the parsing and the filing with 18 checks. Four copiescarrying one deliberate defect each failed it by name.
The fix
The suite is built once and that build is run N times, so every difference between runs is the
tests rather than the tree. Every test whose outcome changed is named with its rate and the panic it
produced, and gets an issue titled
Flaky test: <name>, the title the one hand-filed flake herealready carries. A test that already has an issue gets nothing further, so a weekly run adds only
what is new, whoever opened the first one. A rate fails
nothing: going red over a three-in-thirty flake would teach people to ignore the job, so the exit
code covers only a measurement that did not happen, or an issue meant to be filed that was not. A
pull request touching the script pays only the selftest.
Why the job names both thread counts instead of letting cargo choose
make check-linuxcaps the suite to--test-threads=4and says why: several turn tests stand up amock HTTP server on an ephemeral port, and at the container's default parallelism enough of them
race that a different one fails each run.
make checkand thecheckjob run at cargo's default,which is the configuration that comment calls broken, so the issue asks whether the cap belongs in
more than one target.
A hosted
ubuntu-latestrunner has four cores, so cargo's default there is already the cap andmeasuring it would answer nothing. The job therefore measures at four threads and at sixteen, which
is what
make checkruns as on the machines people run it on. The difference between the two legsis the answer.
Test plan
cargo build --all --locked- passedcargo test --all --locked- passed, 3558 testscargo fmt --all- no changesmake check- passedmake check-spec- 55 specs, 518 clauses, 0 errorsmake check-npm- passedmake check-reviewdog- no findingsmake check-msrv- passedmake check-linux- passed at--test-threads=4contrib/measure-flakes.py --selftest- 18 checks passed--runs 4 --file-issues brave/bravebot --dry-run- reported 50%, wrote nothing