test(agent): cover the confirm gate and approve-all scoping in the Run loop#132
Merged
Conversation
…n loop Builds on the scriptable-LLM seam (#131) to add the two highest-value agent-loop behaviours that still lacked end-to-end coverage — both safety properties exercised through the real gate chain in Run, not in isolation: - ConfirmDenyInLoop: a tool at/above the confirm threshold prompts; a deny blocks execution; the refusal is fed back as a tool_result and the model re-plans to completion. The loop-level analogue of the unit confirm tests. - ApproveAllScoping: the crown-jewel "approve all" property. Approve-all auto-approves subsequent same-or-lower-risk tools in the same turn (the convenience), but must NOT silently escalate — a tool above the approved tier re-prompts. The batch is [Low, Low, Medium] with approve-all on the first Low (ceiling = Low): the second Low is auto-approved (no prompt), the Medium exceeds the ceiling and re-prompts. Using a Low offline tool (tool_search) for the approved legs keeps the assertions purely about gating order with no hardware execution — the escalating Medium tool is denied before it runs. Test-only; no production change (the seam already exists). Verified: task ci green (lint 0 / vet / build / test:full 0 fail / govulncheck clean); task eval 15/15. internal/agent/run_loop_gates_test.go.
This was referenced Jun 26, 2026
xunholy
added a commit
that referenced
this pull request
Jun 26, 2026
…loop (#136) Completes the agent loop's end-to-end safety/recovery coverage. The scriptable-LLM seam (#131) and reflectorFn seam already existed; the operator-steering and failure-recovery behaviours were tested at the unit level (confirm_test, reflexion_test) but never driven through the real multi-turn Run loop. Three integration tests close that gap: - ReviseSkipsToolAndInjectsRevision: a gated tool answered with DecisionRevise must NOT execute; the synthesised tool_result marks it skipped; and the operator's revision is injected into the conversation as a fresh user turn ("please re-plan") so the model re-plans. Asserted against the marshalled history. - ReflexionFiresOnToolError: a failing tool (cbor_decode on bad hex) triggers the reflector and its diagnosis is appended, delimited, to the tool result the model sees. - ReflexionCapPerTurn: four failing tools in one assistant message trigger at most maxReflectionsPerTurn reflections, proving the per-turn counter is shared across every tool in the turn (not one cap per tool). With the dispatch / tool-call-cap / read-only / confirm-deny / approve-all-scoping tests from #131-#132, the loop's safety and recovery behaviours are now covered end-to-end. Test-only; no production change. Verified: task ci green (lint 0 / vet / build / test:full 0 fail / govulncheck clean); task eval 15/15.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Builds on the scriptable-LLM seam (#131) to cover the two highest-value agent-loop behaviours still lacking end-to-end coverage — both safety properties exercised through the real gate chain in
Run, not in isolation.ConfirmDenyInLoop— a tool at/above the confirm threshold prompts; a deny blocks execution; the refusal is fed back as atool_resultand the model re-plans to completion. The loop-level analogue of the unit confirm tests.ApproveAllScoping— the crown-jewel "approve all" property. Approve-all auto-approves subsequent same-or-lower-risk tools in the same turn (the convenience), but must not silently escalate — a tool above the approved tier re-prompts. The batch is[Low, Low, Medium]with approve-all on the first Low (ceiling = Low): the second Low is auto-approved (no prompt), the Medium exceeds the ceiling and re-prompts.Design note: the approve-all test uses a Low offline tool (
tool_search) for the approved legs so the assertions are purely about gating order with zero hardware execution (the escalating Medium tool is denied before it runs) — no panics, no noisy recovery logs, no audit wiring needed.Verification
task cigreen (lint 0 / vet / build /test:full0 fail / govulncheck clean);task eval15/15.Test-only, no production change — no release on its own (rides the next feature release, with #131).