Skip to content

feat(loop): wire ADR 0064 P2 board seam — execution-grounded coder.solve()#61

Merged
mabry1985 merged 3 commits into
mainfrom
feat/coder-board-seam
Jul 3, 2026
Merged

feat(loop): wire ADR 0064 P2 board seam — execution-grounded coder.solve()#61
mabry1985 merged 3 commits into
mainfrom
feat/coder-board-seam

Conversation

@mabry1985

Copy link
Copy Markdown
Member

HIGH-RISK — needs human review before merge

This wires the ADR 0064 P2 board seam: the loop's per-feature dispatch composes
the coder plugin's execution-grounded solve() ladder instead of a bare single
delegate_to(acp) shot, when it's safe to do so. Land as DRAFT.

ADR: https://github.com/protoLabsAI/protoAgent/blob/main/docs/adr/0064-coder-execution-grounded-code-solve.md

What landed

  • coder_seam.py (new) — a thin adapter, not a reimplementation of the ladder.
    It composes plugins.coder.solve (imported lazily/best-effort — this repo
    carries no hard dependency on the coder plugin, matching the "compose delegates,
    don't reimplement" discipline this repo already follows for the ACP/A2A spawn
    primitive):

    • generate(task, feedback) creates a fresh throwaway worktree per candidate
      (feat-<id>.g<n>) and dispatches the ACP coder into it — the "independent-
      parallel acp attempts" the coder plugin's own generate.py docstring already
      flags as "the P2 path."
    • verify(candidate_wt) runs the configured acceptance-test command in that
      worktree
      and reports real pass/fail (never an LLM judge).
    • The winning (test-passing) candidate is promoted to the feature's canonical
      worktree/branch (reusing worktree.promote_worktree, the same primitive
      Max-Mode already uses) so the rest of the drive — fixups, the pre-PR local gate,
      open_pr, the CI bounce, tier escalation — is completely unchanged. Every
      other candidate is reaped.
    • A spent budget with no passing candidate raises SolveExhausted (a
      WorktreeError subclass) rather than opening a PR on an unverified partial.
  • loop.py — one new branch in _drive's dispatch selection (alongside the
    existing keep_wt / max-mode / plain-single-shot branches):

    elif self._use_coder_solve(feature) and not self._ci_feedback.get(fid):
        wt, branch, result = await coder_seam.dispatch(...)

    SolveExhausted was added to the existing capability failure tuple (next to
    NoChangesError/CoderTimeout) — zero other changes to the escalation/retry
    state machine. This is the composition point the ADR asks for: coder.solve()
    searches within the current model tier; a search that never passes is a
    capability failure for THAT tier, so it climbs the coders-map ladder (or blocks)
    exactly like a no-diff dispatch always has. The tier ladder is untouched.

  • store.pyrecord_gens_spent(fid, n) accumulates coder.solve()'s
    generation cost onto a single, replaced gens:<total> label (called once per
    solve() run, win or lose — a failed search still spent gens); _project now
    surfaces it as gens_spent on every feature so portfolio_rollup (a separate
    plugin) can read cost without raw reads, per the ADR's cost-v1 ethos.

  • protoagent.plugin.yaml / README — new config: coder_solve (opt-out valve,
    default on), coder_solve_test_cmd (falls back to local_gate_cmd if blank),
    coder_solve_test_timeout_s, coder_solve_budget, coder_solve_k,
    coder_solve_tree_depth. Version bumped 0.25.00.26.0 (minor).

The exact seam

ready feature (+ acceptance_criteria) → worktree(s) → coder_seam.dispatch():
    greedy → run acceptance tests in worktree → fail?
      best-of-k into k fresh worktrees → execution-select → fail?
        tree-search refine on failing-test feedback
  → promote the passing candidate's worktree → canonical branch
  → (unchanged) fixups → pre-PR local gate → open_pr → in_review

Honest-degrade behavior (non-regressing by construction)

_use_coder_solve / coder_seam.should_use_solve require all three:

  1. the coder plugin is importable (host has it enabled — it ships disabled by
    default, so most repos never reach this at all),
  2. the feature carries acceptance_criteria (the Ready gate's existing oracle),
  3. a runnable acceptance-test command is configured (coder_solve_test_cmd or
    local_gate_cmd) — without one there is genuinely nothing to execute, and per
    the ADR's no-LLM-judge rule we do not fake grounding with a judge instead.

Missing any of the three → today's single delegate_to(acp) shot runs,
byte-for-byte the same as before this PR. A carried-forward re-dispatch (CI bounce /
goal-fix / gate-fix, signalled by _ci_feedback) also skips the seam — same rule
Max-Mode already follows, since that re-dispatch fixes the existing diff with one
coder rather than fanning out fresh candidates.

Deferred (see ADR + notes)

  • Compiling EARS acceptance criteria into generated tests. This is explicitly
    called out in the ADR as the "genuinely new work" and is more than one clean PR.
    The simplest-correct path used here instead: the coder is already prompted
    (_build_prompt) to write tests satisfying the acceptance criteria as part of its
    definition of done; verify() just runs whatever tests exist in the candidate
    worktree via the configured command and gates on the real exit code — honest
    execution, no fabricated grounding, but coarser-grained than a proper EARS→test
    compiler (the Verdict it returns doesn't name individual failing test cases,
    just pass/fail + the raw output tail as feedback).
  • The fusion rung (P3) — not wired here; out of scope for P2.
  • Candidate worktrees from a crash mid-ladder aren't tracked in _inflight until
    coder_seam.dispatch() returns (same pre-existing limitation Max-Mode's
    _dispatch_max_mode already has) — they self-heal on the next attempt via
    create_worktree's idempotent stale-worktree cleanup, but aren't reaped by the
    health sweep in between. A companion improvement, not blocking for this cut.

How to review / test

  1. Read coder_seam.py top-to-bottom — it's the whole seam in one file.
  2. loop.py diff: the one added elif branch in _drive's dispatch selection,
    the one line added to the capability tuple, and the new config fields in
    __init__. Everything else is untouched.
  3. Tests: pytest tests/test_coder_seam.py — the dispatch decision
    (should_use_solve) and dispatch()'s orchestration (winner promoted, losers
    reaped, gens-spent surfaced, SolveExhausted on exhaustion) with plugins.coder.solve
    injected as fakes (no need for the actual coder plugin repo). pytest tests/test_loop.py -k coder_solve — the honest-degrade matrix at the _drive
    level (coder available+oracle → solve path; missing any gate → single shot;
    SolveExhausted → blocked like a capability failure; CI-bounce → skips the seam).
    pytest tests/test_store.py -k gens_spent — the cost-accounting label.
  4. To exercise it live: enable both delegates + coder + project_board, declare
    an acp coder, set project_board.coder_solve_test_cmd (or local_gate_cmd) to
    your repo's real test command, and give a feature acceptance_criteria. Watch
    the loop create feat-<id>.g1, .g2, … candidate worktrees before promoting the
    winner to feat-<id>.

Gate output

$ ruff check .
All checks passed!

$ ruff format --check .
20 files already formatted

$ pytest -q
230 passed, 1 warning in 0.79s

(ruff pinned to 0.15.10 to match .github/workflows/ci.yml.)

🤖 Generated with Claude Code

Co-Authored-By: Claude Opus 4.8 noreply@anthropic.com

mabry1985 and others added 3 commits July 3, 2026 11:40
…lve()

On a fresh build, when the `coder` plugin is importable, the feature has
acceptance criteria, and a runnable acceptance-test command is configured,
the loop now dispatches through coder.solve()'s execution-grounded ladder
(greedy → best-of-k → tree-search) instead of a single delegate_to(acp)
shot — gated on the feature's acceptance tests actually passing in a real
candidate worktree, never an LLM judge.

- coder_seam.py: a thin adapter composing plugins.coder.solve (imported
  lazily/best-effort — no hard dependency on the coder plugin). generate()
  creates a fresh throwaway worktree per candidate and dispatches the ACP
  coder into it; verify() runs the configured acceptance-test command in
  that worktree and reports real pass/fail. The winner is promoted to the
  feature's canonical worktree/branch; losers are reaped. A spent budget
  with no passing candidate raises SolveExhausted (a WorktreeError
  subclass), which the existing capability-failure handling treats exactly
  like NoChangesError/CoderTimeout — so the coders-map tier ladder still
  climbs when search itself stalls.
- Honest degrade: should_use_solve requires all three gates (coder
  importable + acceptance_criteria + a test command); missing any falls
  back to today's single shot, so no existing deployment can regress.
- store.py: record_gens_spent accumulates coder.solve()'s generation cost
  onto a `gens:<total>` label; `_project` surfaces it as `gens_spent` so
  portfolio_rollup can read cost without raw reads.
- Deferred: compiling EARS acceptance criteria into generated tests. The
  coder is already prompted to write tests satisfying the acceptance
  criteria as part of its definition of done; verify() runs whatever tests
  exist via the configured command rather than fabricating them.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
coder.solve() (plugins/coder/solve.py) has no try/except around its injected
generate()/verify() callables — it assumes a candidate attempt only ever fails
its tests, never errors. But a real ACP dispatch can raise for real (CoderTimeout
on one best-of-k candidate, a worktree op erroring), and that propagated straight
out of dispatch() uncaught, leaking every worktree already created for that run:
untracked in the loop's `_inflight` map until dispatch() returns, and invisible
to the health sweep (a `.gN` candidate id isn't a real board feature, so the
sweep's own get_feature() lookup raises and it skips reaping). This was a much
more common trigger than the "crash mid-ladder" case the PR already called out
as a known limitation — a single slow candidate in best-of-k hitting
coder_timeout_s would hit it on any normal run.

Wrap the solve() call so any exception reaps every candidate seen so far,
surfaces the attempted generation count as spent cost (a failed dispatch still
spent the gen), and re-raises the original exception unchanged so the loop's
existing capability-failure classification (retry/escalate/block) still applies.

Also drop the `typing.Optional` usage in this file for the `X | None` style
every other module here already uses under `from __future__ import annotations`,
and hoist the (always-available, stdlib) `importlib` import to the top of the
file with the rest of the module's imports.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
…/coder_solve precedence

Two adversarial-review blockers on the ADR 0064 P2 board seam:

- coder_seam.dispatch()'s two record_gens(...) calls (mid-ladder-raise and
  post-solve()) were unguarded, even though store.record_gens_spent's own
  docstring promises fire-and-forget semantics. A transient BoardError from
  a `br` hiccup would propagate past _drive's capability-failure handling
  into the outer `except BoardError`, which never reaps a worktree — leaking
  an already-verified (or already-reaped) candidate and blocking the feature
  over a bookkeeping label write. Wrapped both call sites in a shared
  _record_gens_best_effort() helper that logs and swallows.

- _use_coder_solve() let coder.solve() preempt Max-Mode purely because the
  separate `coder` plugin became importable, even on a board already
  configured with max_mode_n>1 + local_gate_cmd (the README's own documented
  execution-grounded Max-Mode recipe). Since coder_solve defaults on and its
  test-cmd falls back to local_gate_cmd, that board flips from "always ships
  a best-effort PR" to "blocks outright on an exhausted search" with zero
  config change of its own. coder.solve now only preempts Max-Mode when
  max_mode_n<=1; a board wanting the ladder over Max-Mode must opt in
  explicitly.

Added tests pinning both fixes (record_gens raising on the promoted-winner,
exhausted, and mid-ladder-raise paths; Max-Mode-wins-precedence).
@mabry1985
mabry1985 marked this pull request as ready for review July 3, 2026 19:22
@mabry1985
mabry1985 merged commit 1b323fc into main Jul 3, 2026
1 check passed
@mabry1985
mabry1985 deleted the feat/coder-board-seam branch July 3, 2026 19:23
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant