Skip to content

Measure test determinism rather than guessing which tests flake - #261

Merged
bbondy merged 1 commit into
brave:mainfrom
netzenbot:fix-measure-test-determinism
Sep 13, 2026
Merged

bbondy merged 1 commit into
brave:mainfrom
netzenbot:fix-measure-test-determinism

Conversation

@netzenbot

@netzenbot netzenbot commented Sep 13, 2026 •

Copy link
Copy Markdown
Collaborator

Closes #223

The problem

Nobody can say which tests here are flaky. docs/development/checks.md tells anyone hitting a
failure that does not reproduce to name it and carry on, which is good advice in front of a red
build and also a policy that can absorb a real regression: "it reproduces on the parent, carry on"
is only safe if somebody knows which tests behave that way. The one class anybody can point at,
Text file busy (os error 26), is known because two people happened to file it (#18, #76).

Reproduce

  1. From the repo root, ask which jobs run the suite more than once: grep -rn "cargo test" .github/workflows/
  2. Read the flake paragraph in docs/development/checks.md, then try to answer "which tests, and
    how often?"

Every job runs the suite once, so there is nothing to read: a flake stays a shrug however many runs
it costs. Expected: a list of names with rates.

With this branch, contrib/measure-flakes.py --runs 10 reported 3558 tests reported. Every test agreed with itself across all 10 runs. on a 16 core macOS host. With a test written to lose every
other run it reported flake_demo::a_quiet_test_loses_a_run_now_and_then | 2 | 4 | 50% and its
panic, and kept a test that failed every run out of that table. What it would have filed was checked
against the live issue list: the known flake matched #76 and was left alone.

contrib/measure-flakes.py --selftest pins the parsing and the filing with 18 checks. Four copies
carrying one deliberate defect each failed it by name.

The fix

The suite is built once and that build is run N times, so every difference between runs is the
tests rather than the tree. Every test whose outcome changed is named with its rate and the panic it
produced, and gets an issue titled Flaky test: <name>, the title the one hand-filed flake here
already carries. A test that already has an issue gets nothing further, so a weekly run adds only
what is new, whoever opened the first one. A rate fails
nothing: going red over a three-in-thirty flake would teach people to ignore the job, so the exit
code covers only a measurement that did not happen, or an issue meant to be filed that was not. A
pull request touching the script pays only the selftest.

Why the job names both thread counts instead of letting cargo choose

make check-linux caps the suite to --test-threads=4 and says why: several turn tests stand up a
mock HTTP server on an ephemeral port, and at the container's default parallelism enough of them
race that a different one fails each run. make check and the check job run at cargo's default,
which is the configuration that comment calls broken, so the issue asks whether the cap belongs in
more than one target.

A hosted ubuntu-latest runner has four cores, so cargo's default there is already the cap and
measuring it would answer nothing. The job therefore measures at four threads and at sixteen, which
is what make check runs as on the machines people run it on. The difference between the two legs
is the answer.

Test plan

  • cargo build --all --locked - passed
  • cargo test --all --locked - passed, 3558 tests
  • cargo fmt --all - no changes
  • make check - passed
  • make check-spec - 55 specs, 518 clauses, 0 errors
  • make check-npm - passed
  • make check-reviewdog - no findings
  • make check-msrv - passed
  • make check-linux - passed at --test-threads=4
  • contrib/measure-flakes.py --selftest - 18 checks passed
  • --runs 4 --file-issues brave/bravebot --dry-run - reported 50%, wrote nothing
  • CI passes cleanly

@netzenbot netzenbot self-assigned this Sep 13, 2026
checks.md tells anybody hitting a failure that does not reproduce to name
it and carry on, which is right, and nothing in this repo ever measured
which tests actually behave that way. So the answer to "is that a flake"
was a shrug, and it stayed a shrug however many times the same test lost
a run.

contrib/measure-flakes.py builds the suite once, runs that build N times,
tallies every test's outcome per run, and reports each test whose outcome
changed with the rate it changed at and the panic it produced. Cargo
announces each test binary on stderr while libtest writes outcomes on
stdout, so the two streams are read as one: without the announcement a
name like `a_reply_arrives_before_the_deadline` cannot be told apart from
the same name in another binary.

Every test it names then gets an issue titled `Flaky test: <name>`, which
is the title the one flake anybody here filed by hand already carries, so
a test that already has an issue gets nothing further whoever opened it.
A rate fails nothing: a test failing three runs in thirty is exactly what
the policy above describes, and going red over it would teach people to
ignore the job. The exit code is nonzero only when the measurement did
not happen, or when an issue that was meant to be filed was not, since a
filer that has quietly stopped working looks exactly like a suite that
stopped being flaky.

The new Test determinism workflow is the first scheduled one here. It
runs weekly against the default branch, files what it finds, and leaves
the rates in the job summary. A pull request touching the script runs its
selftest and nothing else, since an hour of suite runs in front of a
review is worthless and a parser that matches nothing would otherwise
report a clean suite forever.

It measures at four threads and at sixteen, both named rather than left
to cargo. A hosted runner has four cores, so cargo's default there is
already the cap `make check-linux` sets, and the wide leg is what `make
check` runs as on the machines people run it on. The difference between
the two legs is what says whether that cap belongs in more than one
target. Only the wide leg files, since both legs measure the same tests
and both filing would race to open two issues for one flake.

Evidence on a 16 core macOS host: ten runs of all 3558 tests at default
parallelism found nothing that disagreed with itself. A test written to
lose every other run was reported at 50 percent with its panic, and what
it would have filed was checked against the live issue list: the known
flake matched brave#76 and was left alone, and only the new name would have
been opened.

Closes brave#223
@netzenbot
netzenbot force-pushed the fix-measure-test-determinism branch from f19a97c to 229884a Compare September 13, 2026 19:16
@bbondy
bbondy marked this pull request as ready for review September 13, 2026 21:06
@bbondy
bbondy merged commit 766f124 into brave:main Sep 13, 2026
17 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

ci: nothing measures test determinism, so a flake is a shrug rather than a named test with a rate

2 participants