feat(bench): a committed census for the tool share of wall clock, and what it says about the 75% - #6503
Merged
Merged
Conversation
#3753 reports tool execution at 75% of agent wall clock on one match, and asks for that number again on a comparable panel "on the same clean-basis method". The method was an uncommitted jq incantation, so the next panel would have produced a figure nobody could hold against the last one. census.py is that method. Wall clock is the first event to the last, per trial; tool time is tool_result.duration_ms over the join run_trace already performs; a share is a ratio of two sums over one named trial set. Run against the match #3753 cites it answers 52.0% over 12 trials, reproducing the 51.8% already published on the issue from the same match. The basis rides on every result because this issue is the case for it. The body says 75% over a clean basis of nine tasks and a later comment says 52% over the twelve that carried events. Neither corrects the other and either one quoted alone describes a population the reader cannot see. Three trial kinds are separated rather than summed. A trace from before the event schema carried ts has tool durations and no wall clock, so it stays out of the numerator as well as the denominator instead of entering as a zero that deflates the share. A trial that recorded nothing is counted apart from that, because a credential that never authenticated writes empty traces and folding them together reports a broken run as an old one. And a task directory without stella-events.jsonl is the other contestant's seat, skipped rather than counted as our failure — walking on the agent/ directory alone read 16 opposing trials as trials of ours that died. Inside the tool time it names two costs the trace can establish with no rig and no spend: a call bash killed at its timeout, and a killed call whose command a later call repeats. The second is the shape pytorch-model-cli died in — an install killed at 120s, the same install killed again at 300s, then a third attempt that ran. Refs #3753
![sourcery-ai[bot]](https://avatars.githubusercontent.com/in/48477?s=60&v=4){kind=small}
Contributor
There was a problem hiding this comment.
Sorry @macanderson, you've used your own review budget of 250,000 diff characters for the last 7 days.
You can request another review in 2 days and 4 hours by commenting @sourcery-ai review. Upgrade to get a review now.
Contributor
Reviewer's GuideIntroduces a committed, offline census for tool execution’s share of wall clock, with explicit trial-basis accounting and diagnostics for timeout/retry waste, plus tests that protect schema, population, and filesystem-boundary distinctions. Sequence diagram for trial filtering and share calculationsequenceDiagram
participant CLI
participant Loader as _load
participant CensusOf as census_of
participant Summarize as summarize
participant Render as render
CLI->>Loader: load_run(root)
Loader-->>CLI: trials with stella-events.jsonl
loop each trial
CLI->>CensusOf: census_of(trial)
CensusOf->>CensusOf: _wall_ms(trial)
CensusOf-->>CLI: TrialCensus
end
CLI->>Summarize: summarize(censuses)
Summarize->>Summarize: exclude untimed trials from wall/tool/model shares
Summarize->>Summarize: keep empty_trials separate
Summarize-->>CLI: Census
CLI->>Render: render(census, label)
Render-->>CLI: basis-labelled report
Flow diagram for the offline trace censusflowchart LR
Root[Match or run directory] --> Load[_load]
Load --> Trials[Trial events]
Trials --> PerTrial[census_of]
PerTrial --> TrialCensus[TrialCensus]
TrialCensus --> Summarize[summarize]
Summarize --> Census[Census totals and shares]
Census --> Render[render]
Render --> Output[Offline text report]
File-Level Changes
Possibly linked issues
Tips and commandsInteracting with Sourcery
Customizing Your ExperienceAccess your dashboard to:
Getting Help
|
macanderson
enabled auto-merge (squash)
September 11, 2026 18:59
2 tasks
macanderson
added
the
closes-nothing
The census rendered `bash 24.48h over 18705 tool calls` on one line under a `tool execution 14.95h` line, which says three wrong things at once. The call count is every tool's, not bash's. The shell total is over every trial while the tool total is over the trials that carry a timestamp, so the smaller number does not contain the larger one. And the indentation read as though it did. I published that misreading before catching it — the count went out on #3753 as "24.48h of bash wall clock in 18,705 calls" when the bash calls number is 13,245. The comment and the pull request body are corrected; this is the render that invited it. Now each section states what it is over. A share needs a clock and is held to the trials that carry one. A duration does not, so a trace on the older schema still contributes its shell time, under its own heading. `bash_calls` sits beside `tool_calls` rather than standing in for it. Refs #3753
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What & why
#3753's remaining definition of done asks for the tool-execution share of wallclock "on the same clean-basis method" as the 75% in its body. That method was an
uncommitted
jqincantation.bench/trace_triage/census.pyis the method writtendown, so the next panel produces a figure a reader can hold against this one.
Wall clock is the first event to the last, per trial. Tool time is
tool_result.duration_msover the joinrun_tracealready performs. A share is aratio of two sums over one named trial set, never a mean of ratios.
Calibration
Run against the match
#3753cites, the committed module answers 52.0% over 12trials. The figure already published on that issue from the same match, by a
separate hand-written join, is 51.8% (6,578s wall, 3,406s tool). The wall
clocks are identical and the tool totals differ by 13s. I did not establish what
that 13s is — the other join is not in the tree — so it is stated as an
unexplained residual rather than attributed.
What it says about the 75%
Pointed at every ArenaBench match on this machine — 294 matches, 341 Stella trials
that carry a wall clock — the share is not 75% and not 52%:
#3753's body, clean basis of 9#3753's later comment, same matchLabelled as a measurement, not a panel. These are the matches that happen to be
on disk, run against many configurations over months, not a controlled panel. What
it establishes is narrow and worth having: the 75% is a property of an
install-heavy task mix, not of the harness. On the corpus the model is the larger
consumer by a wide margin.
The practical consequence: a panel reporting a lower number than 75% has probably
changed task mix, and the census makes that visible because the trial count
travels with the share.
It does not discharge either remaining box on
#3753. Both still need a fundedpanel.
The second cost inside shell time
#6440and#6501are both aimed at blind waiting. The census names a largerbucket beside it, over 24.48h of
bashwall clock in 13,245 bash calls:The second row is time that bought nothing at all: the call was killed, and a later
call in the same trial re-ran it. That is the shape
pytorch-model-clidied in —pip install torchkilled at 120s, the same install killed again at 300s, then athird attempt that ran. 420s of its 984s went into the two attempts that were
thrown away, before the 490s blind wait
#6501declines.No fix here. The remedy is backgrounding, which
#6501argues is a body of workrather than a session, and which
#3753already tracks as its unbuilt directiontwo. The number is here so whoever sizes that work has it.
Three trial kinds kept apart, and two bases kept apart
Each of these was a wrong number before it was a distinction:
tshas tool durations and nowall clock. It stays out of the numerator as well as the denominator, instead of
entering as a zero that deflates the share. 180 of the traces here are this.
authenticated writes empty traces, and folding them in reports a broken run as an
old one — the separation
postmortemputs first, for the same reason.stella-events.jsonlis the other contestant'sseat. Walking on the
agent/directory alone read 16 opposing trials on13f7f2bb533das trials of ours that died.The render also prints the shell figures under their own heading. A share needs a
clock and is held to the trials that carry one; a duration does not, so a trace on
the older schema still contributes its shell time. Printed as one block, the
smaller tool figure read as though it contained the larger shell one.
1aabe7c76is that fix, and it followed me publishing the misreading on
#3753— the bashhours were right and the call count beside them was every tool's.
The witness
main, pass here).bench/trace_triage/tests/test_census.py:test_a_trial_without_a_timestamp_stays_out_of_every_share— the 600s an untimedtrial spent is absent from the share and still visible in
bash_ms.test_a_killed_call_whose_command_comes_back_is_paid_for_twice— the commandspytorch-model-cliactually sent, verbatim.test_a_run_that_never_timed_out_reports_no_waste— quiet on a healthy run.test_the_rendered_census_names_the_basis_of_its_shareandtest_a_share_with_no_wall_clock_behind_it_is_absent_rather_than_zero.test_a_trial_that_recorded_nothing_is_not_reported_as_an_old_schema.test_the_other_contestants_seat_is_not_a_trial_that_recorded_nothing.Offline by construction: every trace is written, never fetched.
The gate
pytest bench/trace_triage/tests/test_census.py— 7 passedruff checkandruff format --checkon both new filesmake prose,make line-citations,scripts/check-file-size.shbench.ymlrunspytest bench/trace_triage/testson every pull request and itsscope filter names
bench/, so this suite is gated in CI rather than only here.No workspace build was run locally, per SCR-001.
Conflict surface
Two new files, nothing shared touched.
#6501addsbench/trace_triage/waits.pyfor blind waits and edits
detectors.pyandtests/test_postmortem.py; thisbranch touches none of those, so the two can land in either order.
Refs #3753