Skip to content

Sweep a hundred times, and stop the mock inventing failures - #267

Merged
bbondy merged 2 commits into
brave:mainfrom
netzenbot:sweep-a-hundred-times
Sep 13, 2026
Merged

bbondy merged 2 commits into
brave:mainfrom
netzenbot:sweep-a-hundred-times

Conversation

@netzenbot

@netzenbot netzenbot commented Sep 13, 2026 •

Copy link
Copy Markdown
Collaborator

The problem

The weekly sweep runs the suite thirty times, which only accounts for a test losing about a tenth of
its runs; under that a clean sweep is worse than a one in twenty coincidence. A tenth is where this suite's flakes were
measured, ten runs each.

None of the nine tests it named failed readably. Six said Connection refused, none of it true of what
went wrong, and three failed in the product's own words: a turn one round short, a note
shown to nobody, output that never reached the planner. All nine were the mock server.

Reproduce

  1. contrib/measure-flakes.py --runs 10 --test-threads 4, in a loaded 16 core container:
    9 tests in crates/agent/tests/turn.rs, one at 4 of 10, six of them Connection refused.
  2. Ask the mock the same thing twice, which is a retry. On the parent commit the second ask
    fails in connect: Os { code: 61, kind: ConnectionRefused }.
  3. Script 205 rounds and leave the first reply without an ending. The parent mock takes the turn to
    204 steps, the assertion the 40% test failed on; with the memo below it reaches 205.

The fix

The chat client resends a request whose reply stopped early, and the mock had one accept per scripted
reply and no memory of its answers. So a resend either arrived at a port whose listener had
gone away with the last reply in it, and a refused connection is classified as permanent, or it was
handed the reply the next round was owed, which shifted a positional script by one and ended the turn
a round early. Six of one, three of the other.

The listener now outlives its script in serve_sequence, in serve_pages and in the one-off server
the binary body test stands up, as it already did in serve_by_marker_meeting. A body answered before
gets that same answer again: only a resend can repeat a body, since every round carries the ones
before it. An attempt the script was told to lose is not remembered, so
serve_sequence_losing_the_first still means what it says. Past the script the answer is
400 and the reason, a status the client does not retry.

The weekly count and the dispatch default go to a hundred, reaching a test losing three runs in a
hundred. A ten run dispatch took about four minutes a leg, build included.

Test plan

  • cargo test -p bravebot-agent --test turn - 258 pass, 2 of them new
  • The new test on the parent commit - fails in connect, errno 61, as the flakes did
  • A 205 round turn whose first reply is cut short - 204 steps on the parent mock, 205 with the memo
  • make check - 3562 tests, exit 0
  • make check-spec - 55 specs, 518 clauses, 0 errors
  • make check-reviewdog - no findings
  • ruby -ryaml parse: dispatch default "100", RUNS reads inputs.runs || '100', timeout 240
  • CI passes cleanly

Thirty runs can only account for a test that loses about a tenth of them,
and a tenth is where the flakes found in this suite were measured, over
ten runs each. A rate that rough sits on either side of that line, so the
weekly job could run its twenty minutes, report that every test agreed
with itself, and have cleared none of the tests it was looking for.

A hundred runs reaches a test losing three in a hundred. A ten run
dispatch took four minutes a leg, nearly all of it the suite rather than
the build, which puts a hundred at about half an hour per leg: a weekly
job's to spend, and well inside the timeout already on it.

Every report already states the rate it reached, so this changes what
that sentence is allowed to claim rather than adding a claim to it. The
same container that named nine tests, one of them failing four runs in
ten, named nothing at all in thirty runs once the host went quiet.
Six tests in crates/agent/tests/turn.rs lost runs on a loaded machine
with `Connection refused`, which explains nothing and is not true of the
thing that actually failed.

The chat client resends a request whose reply stopped early: half a reply
cannot be continued, so the whole request goes again. The mock server
accepted as many connections as its script had replies and then let its
listener drop with the thread that held it. The resend therefore arrived
at a port with nothing behind it, and a refused connection is classified
as permanent, so the turn failed on a socket rather than on whatever cut
the first reply short. Those runs would have passed had the server still
been listening, and the error they reported named neither cause.

The listener now outlives the script in serve_sequence, in serve_pages
and in the one-off server the binary body test stands up, as it already
did in serve_by_marker_meeting. A request whose body has been answered
before gets the same answer again, so a resend is not handed the reply
the next round was going to get: only a resend can repeat a body, since
every round of a turn carries the rounds before it. An attempt the script
was told to lose is not remembered, so serve_sequence_losing_the_first
still means what it says. A request past the end of the script is
answered with 400 and the reason, a status the client does not retry.

serve is three lines delegating to serve_sequence now, having been a copy
of it that accepted once.

Two tests cover the harness itself, because a mock server that fails in
its own right is a suite reporting on the wrong thing.
@netzenbot netzenbot changed the title Sweep a hundred times a week rather than thirty Sweep a hundred times, and stop the mock inventing a refusal Sep 13, 2026
@bbondy
bbondy merged commit b101346 into brave:main Sep 13, 2026
18 checks passed
@netzenbot netzenbot changed the title Sweep a hundred times, and stop the mock inventing a refusal Sweep a hundred times, and stop the mock inventing failures Sep 13, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants