mistake-journal.jsonl

{"id": "LJ-0001", "at": "2026-08-09T14:51:50Z", "population": "producer", "kind": "backgrounded-gate-strands-the-run", "run": "20260809T145150Z", "trace": "runs/20260809T145150Z.jsonl", "what": "The run's first act was `state-load` in a `Bash` call with no `timeout`, so the harness moved it to the background at 120s. The run then armed a `Monitor` on an `until grep -q 'exit=' ...; do sleep 10; done` loop over the backgrounded command's output file, Read that file while it was still empty, wrote \"State-load is running ... monitoring for completion\" and returned. `Monitor` does not block: it arms a background watcher and returns immediately, delivering events as notifications on a LATER turn, and under `claude --print` there is no later turn. The process exited, the watcher died with it, and nothing was ever read. 168 seconds, no state.json, no queue, no work -- and the run reported SUCCESS, because it had arranged the wait exactly as the prompt then prescribed. The rule had been followed; the mechanism the rule named could not do the job.", "evidence": ["turn 1 tool_use Bash: {\"command\": \"pr-review-report state-load --json > /home/gildlab/code/scratch/20260809T145150Z-3791269/state.json 2>/home/gildlab/code/scratch/20260809T145150Z-3791269/state.err; echo \\\"exit=$?\\\"; jq '{byAction: .fleet.byAction, ...}' .../state.json\", \"description\": \"Load pipeline state\"} -- no `timeout` argument, so the call inherited the 120s default.", "tool_result: \"Command did not complete within its 120s timeout and was moved to the background (ID: bb7tt82ta). Output is being written to: /tmp/claude-1000/-home-gildlab-code/0fbab6e7-ca74-4ac6-97e2-cb29016e5bb9/tasks/bb7tt82ta.output. You will be notified when it completes.\"", "turn 2 tool_use Monitor: {\"command\": \"until grep -q 'exit=' /tmp/.../tasks/bb7tt82ta.output 2>/dev/null; do sleep 10; done; echo \\\"STATE-LOAD-DONE\\\"\", \"description\": \"wait for state-load to finish\", \"timeout_ms\": 600000, \"persistent\": false}", "tool_result: \"Monitor started (task b73n0v1az, timeout 600000ms). You will be notified on each event. Keep working -- do not poll or sleep. Events may arrive while you are waiting for the user -- an event is not their reply.\"", "turn 3 tool_use Read of the backgrounded command's output file returned: \"<system-reminder>Warning: the file exists but the contents are empty.</system-reminder>\"", "final assistant text: \"State-load is running (large org-wide read); monitoring for completion.\"", "result event: {\"type\":\"result\",\"subtype\":\"success\",\"is_error\":false,\"num_turns\":5,\"duration_ms\":168081} -- the run's own record of itself says it succeeded."], "rule": {"landedAt": "2026-08-09T15:14:58Z", "site": "campaign-prompt.txt -- the shell-shapes paragraph (WAITING IS A FOREGROUND `Bash` CALL) and step 2 (STATE-LOAD)", "issue": "https://github.com/rainlanguage/issue-pr-cron/issues/249"}}
{"id": "LJ-0002", "at": "2026-08-09T15:52:41Z", "population": "producer", "kind": "fact-asserted-into-a-dispatch-brief-without-reading-it", "run": "20260809T155241Z", "trace": "runs/20260809T155241Z.jsonl", "what": "The producer told two dispatched workers that their PRs already carried a prior `[3b-attempt]` commit, so the step-3b back-off rule was live. Neither did. The `state-load` rows both briefs were built from carried `has3bAttempt: false` on both PRs, and the producer had those rows in context when it wrote the briefs. Both workers checked the commit history themselves and caught it, and it changed neither route -- but the premise was invented, twice, in the same run, about a field the tool had already answered. Nothing in the prompt addresses asserting a state-load field into a brief without reading it back; the run caught itself only in its own closing report.", "evidence": ["dispatch brief for rainlanguage/rain.factory#51: \"Note: this PR already carries a prior `[3b-attempt]` commit, so the step-3b back-off rule is live -- check its commit history first and if a previous attempt already ...\"", "dispatch brief for rainlanguage/rain.factory.deploy#7: \"This PR already carries a prior `[3b-attempt]` commit, so the step-3b back-off rule is live per failing check -- read the ...\"", "the rows both briefs were built from: `rainlanguage/rain.factory\\t51\\t{\"requiresRedeploy\":false,\"deployDoneAtHead\":false,\"designFlicked\":false,\"handedOff\":false,\"has3bAttempt\":false,\"screenshotSettled\":false,\"uiTouch\":\"no\"}` and `rainlanguage/rain.factory.deploy\\t7\\t{...,\"has3bAttempt\":false,...}`", "the run's own correction, mid-run: \"Both `has3bAttempt` were `false` -- the back-off premise in those two dispatches was mine, not the classifier's. It didn't change either route, but it was a fact I asserted without reading.\"", "and again in the run's final report: \"One correction on me: I told both 3b workers their PRs already carried a `[3b-attempt]` commit. Neither did -- `state-load` had `has3bAttempt: false` on both rows and I asserted it without reading them.\""], "rule": null}
{"id": "LJ-0003", "at": "2026-08-10T09:00:00Z", "population": "interactive", "kind": "ci-failure-attributed-to-the-wrong-pre-commit-hook", "run": null, "trace": null, "what": "Two separate agents, in one session, attributed the same rainix pre-commit failure over a markdown file to `prettier-rainix`. `prettier-rainix` cannot touch markdown: its `types_or` is svelte/ts/javascript/json. Markdown in rainix's `rust-shell` belongs to `denofmt`, whose `excludes` list is exactly those four types and nothing else. Both agents then reasoned about the fix from the wrong hook's behaviour. No run trace exists because this was interactive work in this repo, not a cron run -- which is itself the reason the journal's population is wider than the two runners: the same model makes the same class of mistake against the same documents whether a cron started it or a human did.", "evidence": ["rainix flake.nix at 53e96a7d0a97d7c7c75c3b2412521324776fdac6 defines the hook as `prettier-rainix = { enable = true; name = \"prettier-rainix\"; entry = ...; types_or = [ \"svelte\" \"ts\" \"javascript\" \"json\" ]; };` -- markdown is not among its types, so it never sees a .md file.", "the same file defines `denofmt = { enable = true; excludes = [ \".*\\\\.ts$\" \".*\\\\.js$\" \".*\\\\.json$\" \".*\\\\.svelte$\" ]; };` -- an exclude-only list, so every other extension including .md is denofmt's.", "the misattribution: both agents named `prettier-rainix` as the hook that rewrote a markdown file under `nix develop github:rainlanguage/rainix/53e96a7d0a97d7c7c75c3b2412521324776fdac6#rust-shell -c pre-commit run --all-files`, and neither read the hook definitions before saying so."], "rule": null}
{"id": "LJ-0004", "at": "2026-08-10T10:23:25Z", "population": "interactive", "kind": "foreground-call-prescribed-for-work-that-outlives-it", "run": "20260810T102325Z", "trace": "review-runs/20260810T102325Z.jsonl", "what": "`/human-fsm:observe-run` step 1 said to start `force-run` in a **foreground** `Bash` call because \"it blocks for the whole run, which is the point\". A foreground `Bash` call in this harness is moved to the background at a 600-second ceiling and hands back nothing of the stream when it is, and the work it is pointed at outlives that by minutes to tens of minutes: on 2026-08-10 producer `20260810T083614Z` took 1,967,592 ms (32m 48s) and vetter `20260810T091521Z` 1,350,441 ms (22m 30s). So for the common case the instruction could not be honoured at all, and what it did buy was a blank screen: a foreground call shows nothing until it returns, so a run that is working and a run that has hung are the same picture. The caller compounded it by piping the stream through `tail`, which holds every byte until the writer exits, so even after the ceiling backgrounded the call its output file was still empty ten minutes in. On the next attempt, 7m 07s of silence read as a hang: the human rejected the call, killing vetter run `20260810T102325Z` -- which had passed the fast-forward gate, walked both policy stops, dispatched three `pr-auditor` sub-agents and was 52 tool calls deep and working. Its trace survives truncated; the run does not, and it never wrote a `final` row. The counter-example was in the same session an hour and a half earlier and nobody read it as one: the producer run was started in a BACKGROUND call -- not what step 1 said -- and its output file was readable 6 seconds later with the fast-forward verdict, the `log:` path and the runner invocation already in it.", "evidence": ["the step, before this rule (`plugins/human-fsm/commands/observe-run.md`, step 1): \"Run it in a **foreground** `Bash` call. It blocks for the whole run, which is the point; wrapping it in anything that returns immediately abandons the wait.\"", "2026-08-10T09:15:16Z tool_use Bash: {\"command\": \"cd /home/gildlab/issue-pr-cron && pr-review-report force-run vetter --install-dir /home/gildlab/issue-pr-cron 2>&1 | tail -40\", \"timeout\": 600000} -- foreground, and piped.", "2026-08-10T09:25:19Z tool_result: \"Command did not complete within its 600s timeout and was moved to the background (ID: bflmjrq1m). Output is being written to: /tmp/claude-1000/-home-gildlab-code/df06f777-5716-4222-9132-d58a39dc2642/tasks/bflmjrq1m.output.\" -- then a Read of that file 5 seconds later returned \"<system-reminder>Warning: the file exists but the contents are empty.</system-reminder>\". Ten minutes of run, zero bytes, because `tail` was holding them; the file reached its final 6648 bytes only at 09:37, when the run exited.", "2026-08-10T10:21:23Z tool_use Bash: the same invocation with `| tail -50`, foreground, timeout 600000 -- and 2026-08-10T10:28:31Z tool_result: \"The user doesn't want to proceed with this tool use. The tool use was rejected\". 7m 07s, before the 600s ceiling had even fired. 18 seconds later the human wrote: \"hanging nearly 10 mins\", and at 10:29:58Z: \"rerun without the tail pipe\".", "`review.log` for the killed run: \"2026-08-10T10:23:25Z FORCED past the review-DISABLED kill switch (--force)\", \"2026-08-10T10:23:25Z FORCED past the usage-gate PAUSE (--force): PAUSE: 16% used vs 13% linear-by-now toward reset 2026-08-16T11:59:59.573735+00:00 (endpoint)\", then \"review run START\" -- and no `END`/`SKIP`/`ABORT` line for it anywhere; the next line in the file is the following run's force at 10:30:08Z.", "`metrics/runs.jsonl` holds a `boot` and one `usage` row for `20260810T102325Z` and no `final` row, so the run contributed no duration, no cost and no outcome. Its neighbours did: `\"durationMs\": 1967592` (producer `20260810T083614Z`), `\"durationMs\": 1350441` (vetter `20260810T091521Z`), `\"durationMs\": 426315` (vetter `20260810T103008Z`).", "`pr-review-report token-profile review-runs/20260810T102325Z.jsonl` over the truncated trace: \"turns 23 / tool calls 52 / context peak 87548\". The trace holds three `Agent` dispatches -- `Audit cyclo.site#385`, `Audit cyclo.site#388`, `Audit cyclo.site#389` -- with a matching `lens` row for each in `review-runs/20260810T102325Z.lens`.", "the counter-example, 2026-08-10T08:36:08Z tool_use Bash: the producer force-run with `run_in_background: true`; it returned in 2 seconds naming its output file, and a Read of that file at 08:36:14Z -- 6 seconds after the call -- already held \"install dir /home/gildlab/issue-pr-cron already current at c80801b\", \"log: /home/gildlab/issue-pr-cron/campaign.log (watch-run reattaches here)\" and \"running: nix run git+file:///home/gildlab/issue-pr-cron#campaign-run -- --force\". That run then ran 32m 48s to a clean exit, unwatched and unkilled."], "rule": {"landedAt": "2026-08-10T12:31:51Z", "site": "plugins/human-fsm/commands/observe-run.md -- step 1 (Force the run, and watch it): a background `Bash` call read through its own output file, nothing buffering between the stream and the reader, and `log:` + `watch-run` as the route back", "issue": "https://github.com/rainlanguage/issue-pr-cron/issues/260"}}
{"id": "LJ-0005", "at": "2026-08-10T09:15:21Z", "population": "vetter", "kind": "inline-audit-concentrates-context", "run": "20260810T091521Z", "trace": "review-runs/20260810T091521Z.jsonl", "what": "Step 2's audit lens is deep source reading, and WHERE it happens decides what a run costs, for a reason that is a property of this harness rather than of the work: a sub-agent carries its OWN context and that context dies with it, while this main loop re-reads its entire history on every turn, so auditing inline concentrates the whole run in one monotonically growing context and pays for it again at every call. Measured on 20260810T091521Z, three PRs audited entirely inline: 72 tool calls, 223,804 cached tokens read PER CALL, a context peaking at 417,832, $27.97 — against the producer's $5.95 for the same 3-item budget over 116 calls at ~30,000 per call, because it dispatches. `campaign-prompt.txt` records that same asymmetry as 75,000 per call dispatching against 264,000 inline, and 224k is the inline figure; 63% of that run's tool-result bytes were the Read/Grep/Glob of one PR still being re-read while the next was judged.", "evidence": ["token-profile review-runs/20260810T091521Z.jsonl (inline): 72 tool calls, 223,804 cached tokens read per call, context peak 417,832, $27.97", "token-profile review-runs/20260810T123636Z.jsonl (first run under the fan-out): 79 tool calls, 41,361 cached tokens read per call, context peak 121,146", "campaign-prompt.txt benchmark line: 75,000 per call dispatching (20260802T130003Z) vs 264,000 inline (20260804T114433Z)"], "rule": {"landedAt": "2026-08-10T10:07:55Z", "site": "review-prompt.txt -- the FAN OUT paragraph (ONE `pr-auditor` PER PR, AND THE VERDICT STAYS YOURS)", "issue": "https://github.com/rainlanguage/issue-pr-cron/issues/257"}}