Watch nightly-bench for ten nights after the loop-gate fix

[macanderson]

Problem

nightly-bench gates on loop health, and #5613 fixed the classification that
held it red: a steering warning the turn survived is no longer scored as a
stuck loop. That fix is proved by two unit tests and by a census of the recorded
streams from four red nights. A dispatched run on the fix branch
(33973144575) is green with LOOP: 0/4 broken, but no detection fired in any
of its four streams, so that run alone cannot tell the new code from the old.

One run is not the acceptance test the original issue asked for. Its last
definition-of-done item was "nightly-bench runs green ten consecutive nights
with LOOP: 4/4", which cannot be ticked before the merge that produces those
nights, so it moves here.

What to do

Watch .github/workflows/nightly-bench.yml for ten consecutive scheduled runs
after the merge of #5613.

• Green with LOOP: 4/4 on all ten: close this, and say so on nightly-bench has been red for two weeks: the loop gate keeps catching a real stuck loop #5613.
• A red night: read its artifact before concluding anything. Download
  loop-bench-<run id> and open events/<task>__<id>.jsonl, then count the
  loop_detected events by their aborted field. aborted: true is a real
  stuck loop and the row is correct — that is an engine finding, and it gets
  its own issue. aborted: false alone must no longer produce STUCK-LOOP; if
  it does, the fix regressed and this issue is where that is reported.

gh api repos/macanderson/stella/actions/runs/<run id>/artifacts
gh api repos/macanderson/stella/actions/artifacts/<artifact id>/zip > a.zip

A CRASHED night is a different signal and is tracked separately — see the
harbor-timeout issue linked from #5613.

Files

• .github/workflows/nightly-bench.yml — the workflow and its pinned task set.
• bench/loop-bench/src/lib.rs — the verdict vocabulary and the two counters
  (loop_detected, loop_aborted).

Definition of done

• Ten consecutive scheduled nightly-bench runs after nightly-bench has been red for two weeks: the loop gate keeps catching a real stuck loop #5613 merged are
  listed here with their run ids and their LOOP: line.
• Any red night among them is explained from its own artifact, and the
  explanation names whether a detection aborted.
• No ceiling was raised and no loop-health arm was weakened.

[macanderson]

Triage: P3, area:bench, area:ci, use-model:cheap, pain:accuracy, pain:ci-time — passive ten-night watch of a merged loop-detection fix; mechanical log reading, low urgency, but the outcome can still surface a real regression or a new engine finding.

[macanderson]

Night observation — 2026-09-08, and the first red one in this watch.

Run 34211343238, head 5887b024, exit 1 (loop health), against the previous night's green run 34110771039 at a8af30fc.

	09-07 green	09-08 red
cobol-modernization	ran (unsolved), 26 calls	solved, 28 calls
fix-git	solved, 36 calls	solved, 52 calls
overfull-hbox	CRASHED (AgentTimeoutError, 750s)	ran (unsolved), 47 calls
prove-plus-comm	solved, 36 calls / 33 tools	STUCK-LOOP, 57 calls / 56 tools
LOOP	0/4 broken	1/4 broken
REWARD	2/4	2/4
spend	$0.06	$0.07

The solve rate did not move. The mechanism metric did, which is the one this gate exists to read.

prove-plus-comm's verdict line:

prove-plus-comm              STUCK-LOOP     57     56    0.02  0.0
  |- stuck-loop detected (persisted after a steering warning): the same `edit_file` call with ...
  |- steered off 2 detected loop(s); the turn carried on
  |- crashed: DeliberateStopError: Stella exited with code 3: a deliberate stop (loop escalation, budge...

So the loop detector fired, steered twice, the model returned to the same edit_file call anyway, and the escalation stopped the turn. That is the machinery working — the question this watch exists for is whether it should have had to.

What I am not claiming. This is n=1. A single task flipping across one night is exactly the resolution a four-task panel cannot settle, and 49 commits sit between the two runs, so a bisect over them would be confounded. One of those 49 is #6441, which added the 8 exit code — but 8 needs the killed trials to outnumber the live ones, and this run was 2 and 2, so what fired is the pre-existing loop-health arm rather than the new rule.

The cheap control before anyone bisects, per AGENTS.md: re-run prove-plus-comm alone against a8af30fc's binary. One trial either establishes the regression or ends the investigation, and it costs about two cents.

Worth reading the other direction too. overfull-hbox went from a 750-second AgentTimeoutError to running to completion in 47 calls, and cobol-modernization from unsolved to solved. The night was not uniformly worse — one task got materially worse while two got better, which is another reason not to read a regression off it alone.

[macanderson]

Red night explained: 2026-09-09, run 34336488244

LOOP: 1/4 broken REWARD: 2/4 solved SPEND: $0.05 — verdicts
CRASHED=1 STUCK-LOOP=1 solved=2.

Read from the artifact (loop-bench-34336488244, id 10098814209) as this issue
asks, counting loop_detected events by their aborted field:

Task	Verdict	loop_detected events	Did a detection abort?
cobol-modernization	STUCK-LOOP	2	yes — one false, then one true
overfull-hbox	CRASHED	1	no — false only
fix-git	solved	0	—
prove-plus-comm	solved	0	—

The fix has not regressed, and this night shows it in both directions.

• cobol-modernization earns its STUCK-LOOP row: a detection with
  aborted: true ended the turn with DeliberateStopError exit 3. By this
  issue's own rule that is a real stuck loop and an engine finding.
• overfull-hbox is the negative control that matters. It carries an
  aborted: false detection and did not get a STUCK-LOOP verdict — its
  CRASHED row is the harbor AgentTimeoutError at 750s, tracked separately.
  Before the fix, that unaborted detection alone would have produced
  STUCK-LOOP. It did not.

So the loop gate is behaving as designed, and the red is the gate reporting a
genuine engine defect rather than misreading a steering warning.

The engine finding now has its own issue, #6480: the steering warning holds for
about thirty seconds and nine varied tool calls, then the agent returns to the
exact command the warning quoted and repeats it three times, which aborts the
turn. The full ordered sequence is written out there.

On the ten-night count

This night does not count toward the ten, since it is not green. For the
record, the scheduled runs since #5613 merged:

Date (UTC)	Run	Result
2026-09-06	34024649792	success
2026-09-07	34110771039	success
2026-09-08	34211343238	failure
2026-09-09	34336488244	failure — explained above

I have not read the 2026-09-08 artifact. Its LOOP: line is not quoted here
because I did not open it, and this issue asks for the line rather than a guess
at it.