Repository navigation
Sweep a hundred times, and stop the mock inventing failures - #267
Merged
Merged
Conversation
Thirty runs can only account for a test that loses about a tenth of them, and a tenth is where the flakes found in this suite were measured, over ten runs each. A rate that rough sits on either side of that line, so the weekly job could run its twenty minutes, report that every test agreed with itself, and have cleared none of the tests it was looking for. A hundred runs reaches a test losing three in a hundred. A ten run dispatch took four minutes a leg, nearly all of it the suite rather than the build, which puts a hundred at about half an hour per leg: a weekly job's to spend, and well inside the timeout already on it. Every report already states the rate it reached, so this changes what that sentence is allowed to claim rather than adding a claim to it. The same container that named nine tests, one of them failing four runs in ten, named nothing at all in thirty runs once the host went quiet.
6 of 7 tasks
Six tests in crates/agent/tests/turn.rs lost runs on a loaded machine with `Connection refused`, which explains nothing and is not true of the thing that actually failed. The chat client resends a request whose reply stopped early: half a reply cannot be continued, so the whole request goes again. The mock server accepted as many connections as its script had replies and then let its listener drop with the thread that held it. The resend therefore arrived at a port with nothing behind it, and a refused connection is classified as permanent, so the turn failed on a socket rather than on whatever cut the first reply short. Those runs would have passed had the server still been listening, and the error they reported named neither cause. The listener now outlives the script in serve_sequence, in serve_pages and in the one-off server the binary body test stands up, as it already did in serve_by_marker_meeting. A request whose body has been answered before gets the same answer again, so a resend is not handed the reply the next round was going to get: only a resend can repeat a body, since every round of a turn carries the rounds before it. An attempt the script was told to lose is not remembered, so serve_sequence_losing_the_first still means what it says. A request past the end of the script is answered with 400 and the reason, a status the client does not retry. serve is three lines delegating to serve_sequence now, having been a copy of it that accepted once. Two tests cover the harness itself, because a mock server that fails in its own right is a suite reporting on the wrong thing.
bbondy
approved these changes
Sep 13, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The problem
The weekly sweep runs the suite thirty times, which only accounts for a test losing about a tenth of
its runs; under that a clean sweep is worse than a one in twenty coincidence. A tenth is where this suite's flakes were
measured, ten runs each.
None of the nine tests it named failed readably. Six said
Connection refused, none of it true of whatwent wrong, and three failed in the product's own words: a turn one round short, a note
shown to nobody, output that never reached the planner. All nine were the mock server.
Reproduce
contrib/measure-flakes.py --runs 10 --test-threads 4, in a loaded 16 core container:9 tests in
crates/agent/tests/turn.rs, one at 4 of 10, six of themConnection refused.fails in
connect:Os { code: 61, kind: ConnectionRefused }.204 steps, the assertion the 40% test failed on; with the memo below it reaches 205.
The fix
The chat client resends a request whose reply stopped early, and the mock had one accept per scripted
reply and no memory of its answers. So a resend either arrived at a port whose listener had
gone away with the last reply in it, and a refused connection is classified as permanent, or it was
handed the reply the next round was owed, which shifted a positional script by one and ended the turn
a round early. Six of one, three of the other.
The listener now outlives its script in
serve_sequence, inserve_pagesand in the one-off serverthe binary body test stands up, as it already did in
serve_by_marker_meeting. A body answered beforegets that same answer again: only a resend can repeat a body, since every round carries the ones
before it. An attempt the script was told to lose is not remembered, so
serve_sequence_losing_the_firststill means what it says. Past the script the answer is400 and the reason, a status the client does not retry.
The weekly count and the dispatch default go to a hundred, reaching a test losing three runs in a
hundred. A ten run dispatch took about four minutes a leg, build included.
Test plan
cargo test -p bravebot-agent --test turn- 258 pass, 2 of them newconnect, errno 61, as the flakes didmake check- 3562 tests, exit 0make check-spec- 55 specs, 518 clauses, 0 errorsmake check-reviewdog- no findingsruby -ryamlparse: dispatch default"100",RUNSreadsinputs.runs || '100', timeout 240