Skip to content

Commit 9dd3700

Browse files
bradygasterclaudeCopilot
authored
test(backend): run the golden trace live on the claude-agent-sdk harness (#2441) (#2452)
* test(backend): run the golden trace live on the claude-agent-sdk harness golden-trace.json (decoupling §14.4, G1–G9) previously ran only against a fake harness. A live variant now runs it on the real Claude Agent SDK behind SQUAD_REAL_CLI_TESTS=1 and an Anthropic API key: a real Bash approval survives a client disconnect and cursor reconnect, the broker approves it and the command runs (the first live approve), then a cancel ends the turn cancelled, with the fixture's backend events in order. Passed 2026-11-10 (Agent SDK 0.3.317, 8.1 s). Recorded in the host feature matrix and decoupling spec §12. Env-gated in CI by the existing reconciliation rule. Refs #2441 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * test(backend): tighten live golden trace review assertions Part of #2441. Address existing feedback on PR #2452: verify replay continuity and cursor suffixes, approve only the exact foreground command, require prompt harness cancellation, and cancel/settle on failure. Qualify historical smoke evidence pending a strengthened live rerun. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: c8c234f1-ee3b-49e1-9406-11d4a2238c23 * test(backend): validate golden Bash execution options Part of #2441. Complete PR #2452's cancellation finding with test-owned full-input interception, explicit long foreground timeout and hash binding before approval, and deterministic short-timeout/background/default negatives plus an abort-honouring control. Keep strengthened live promotion evidence pending. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: c8c234f1-ee3b-49e1-9406-11d4a2238c23 --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: c8c234f1-ee3b-49e1-9406-11d4a2238c23
1 parent 308a7ab commit 9dd3700

4 files changed

Lines changed: 262 additions & 6 deletions

File tree

‎docs/specification/client-backend-decoupling-v0.2.md‎

Lines changed: 13 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -518,7 +518,19 @@ message.accepted(runId, seq)
518518
`client.*` entries are client-side steps that produce no backend event. The
519519
trace is executed end to end against the file-backed local backend and the
520520
durable-http client in `test/squad-20-all-scenarios.e2e.test.ts`, with the
521-
same canonical history on both.
521+
same canonical history on both. A live variant uses the real `claude-agent-sdk` harness
522+
(`test/claude-agent-sdk-harness.test.ts`, behind `SQUAD_REAL_CLI_TESTS=1` and an
523+
Anthropic API key): it checks exact event replay and cursor suffixes, approves
524+
only the exact command after test-owned interception verifies the full hashed
525+
Bash input (`run_in_background: false`, explicit `timeout: 120000`), then requires
526+
cancellation to settle the harness within 10 seconds of its start signal, well
527+
below its 60-second sleep and 120-second tool timeout. Deterministic negative
528+
cases reject short timeouts (including an abort-ignoring transport), background
529+
execution and implicit timeout defaults before approval.
530+
The original smoke run passed on 2026-11-10, but did not prove replay continuity
531+
or prompt harness cancellation. The strengthened live trace awaits a new
532+
credentialed run; the earlier smoke result is not promotion evidence for those
533+
invariants.
522534

523535
This trace captures the core invariants of the split: stable `runId`
524536
cancellation, monotonic `seq` ordering, durable approval versioning, resumable

‎docs/src/content/docs/reference/host-feature-matrix.md‎

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -972,7 +972,7 @@ Depends on: **c** · docs: [concepts/squad-2-0-architecture](/squad/docs/concept
972972
| Host | Capability | Validation | Mechanism | Evidence |
973973
|------|------------|------------|-----------|----------|
974974
| `copilot` | ✅ supported | 🖥️ validated-live | Opt-in tools over the same MCP server; backend semantics are host-independent. The `copilot-sdk` harness (`CopilotSdkHarness`) executes turns on the Copilot SDK; opt-in on the bridge via `SQUAD_BACKEND_HARNESSES=copilot-sdk`. _Live: plain turns through the broker on the real Copilot backend (2026-11-09; again 2026-11-10 after event correlation, including with a harness-only config directory). Approvals are covered by the shared contract, not yet live._ | `validated-test:test/squad-20-real-clients-stdio.e2e.test.ts::completes the MCP handshake and lists squad_state_* and all squad_backend_* tools when opted in`<br>`validated-test:test/squad-20-all-scenarios.e2e.test.ts::Scenario 4: DurableHttpSquadBackend full HTTP lifecycle`<br>`validated-test:test/squad-20-all-scenarios.e2e.test.ts::Full non-bypass journey: host adapter → broker → harness → approval → restart/reconnect`<br>`validated-test:test/harness-contract.test.ts::*`<br>`validated-test:test/copilot-sdk-harness.test.ts::completes a turn through the broker`<br>`validated-test:test/copilot-sdk-harness.test.ts::completes a turn with a harness-only Copilot config directory (login still works)`<br>`validated-manual:lab:copilot-sdk-harness@2026-11-10` |
975-
| `claude` | ✅ supported | 🖥️ validated-live | Opt-in tools over the same MCP server; backend semantics are host-independent. The `claude-agent-sdk` harness (`ClaudeAgentSdkHarness`) executes turns on the Claude Agent SDK when the optional peer is installed. _Live (2026-11-10): one turn through the broker, and a write command approved-or-denied through the broker (denied: not run). Claude Code runs some read-only commands without asking, so those never reach the broker._ | `validated-test:test/squad-20-real-clients-stdio.e2e.test.ts::completes the MCP handshake and lists squad_state_* and all squad_backend_* tools when opted in`<br>`validated-test:test/squad-20-all-scenarios.e2e.test.ts::Scenario 4: DurableHttpSquadBackend full HTTP lifecycle`<br>`validated-test:test/squad-20-all-scenarios.e2e.test.ts::Full non-bypass journey: host adapter → broker → harness → approval → restart/reconnect`<br>`validated-test:test/harness-contract.test.ts::*`<br>`validated-test:test/claude-agent-sdk-harness.test.ts::*`<br>`validated-test:test/claude-agent-sdk-harness.test.ts::routes a real tool call through broker approval and honours a deny`<br>`validated-manual:lab:claude-agent-sdk-harness@2026-11-10` |
975+
| `claude` | ✅ supported | 🖥️ validated-live | Opt-in tools over the same MCP server; backend semantics are host-independent. The `claude-agent-sdk` harness (`ClaudeAgentSdkHarness`) executes turns on the Claude Agent SDK when the optional peer is installed. _Live (2026-11-10): a plain turn and a write denied through the broker (not run). The original golden-trace smoke run approved a real tool call after reconnect, but did not prove replay continuity or prompt harness cancellation. The strengthened trace (exact command and full hashed Bash input validation, explicit 120-second foreground tool timeout, cursor suffix and replay checks, 10-second cancellation deadline) awaits a new credentialed live run and is not yet promotion evidence. Deterministic negatives reject short-timeout/abort-ignoring transport, background execution and implicit timeouts before approval. Claude Code runs some read-only commands without asking, so those never reach the broker._ | `validated-test:test/squad-20-real-clients-stdio.e2e.test.ts::completes the MCP handshake and lists squad_state_* and all squad_backend_* tools when opted in`<br>`validated-test:test/squad-20-all-scenarios.e2e.test.ts::Scenario 4: DurableHttpSquadBackend full HTTP lifecycle`<br>`validated-test:test/squad-20-all-scenarios.e2e.test.ts::Full non-bypass journey: host adapter → broker → harness → approval → restart/reconnect`<br>`validated-test:test/harness-contract.test.ts::*`<br>`validated-test:test/claude-agent-sdk-harness.test.ts::*`<br>`validated-test:test/claude-agent-sdk-harness.test.ts::routes a real tool call through broker approval and honours a deny`<br>`validated-manual:lab:claude-agent-sdk-harness@2026-11-10`<br>`validated-test:test/claude-agent-sdk-harness.test.ts::runs G1–G9 on claude-agent-sdk: approve after reconnect, then cancel` |
976976
| `kiro` | ✅ supported | 🧪 validated-test | Opt-in tools over the same MCP server; backend semantics are host-independent. | `validated-test:test/squad-20-real-clients-stdio.e2e.test.ts::completes the MCP handshake and lists squad_state_* and all squad_backend_* tools when opted in`<br>`validated-test:test/squad-20-all-scenarios.e2e.test.ts::Scenario 4: DurableHttpSquadBackend full HTTP lifecycle`<br>`validated-test:test/squad-20-all-scenarios.e2e.test.ts::Full non-bypass journey: host adapter → broker → harness → approval → restart/reconnect` |
977977
| `opencode` | ✅ supported | 🧪 validated-test | Opt-in tools over the same MCP server; backend semantics are host-independent. | `validated-test:test/squad-20-real-clients-stdio.e2e.test.ts::completes the MCP handshake and lists squad_state_* and all squad_backend_* tools when opted in`<br>`validated-test:test/squad-20-all-scenarios.e2e.test.ts::Scenario 4: DurableHttpSquadBackend full HTTP lifecycle`<br>`validated-test:test/squad-20-all-scenarios.e2e.test.ts::Full non-bypass journey: host adapter → broker → harness → approval → restart/reconnect` |
978978
| `cursor` | ✅ supported | 🧪 validated-test | Opt-in tools over the same MCP server; backend semantics are host-independent. | `validated-test:test/squad-20-real-clients-stdio.e2e.test.ts::completes the MCP handshake and lists squad_state_* and all squad_backend_* tools when opted in`<br>`validated-test:test/squad-20-all-scenarios.e2e.test.ts::Scenario 4: DurableHttpSquadBackend full HTTP lifecycle`<br>`validated-test:test/squad-20-all-scenarios.e2e.test.ts::Full non-bypass journey: host adapter → broker → harness → approval → restart/reconnect` |
@@ -1055,6 +1055,7 @@ Headless real-CLI runs behind the `validated-manual:lab:<scenario>` evidence ent
10551055
| 2026-11-09 | `copilot` | copilot-sdk-harness | ✅ pass | — | copilot-sdk HarnessAdapter (@github/copilot-sdk 1.0.19, signed-in Copilot user) completed one turn through LocalSquadBackend: "Reply with exactly the word: pong" → completed, transcript ends in pong (4.4 s). Plain text only; no tool approval in this run. |
10561056
| 2026-11-10 | `claude` | claude-agent-sdk-harness | ✅ pass | `Bash` | claude-agent-sdk HarnessAdapter (@anthropic-ai/claude-agent-sdk 0.3.317, Anthropic API key) through LocalSquadBackend: a plain turn completed (6.7 s); a Bash `touch` reached the broker as an approval, was denied, ended the turn cancelled, and the file was not created (6.1 s). A read-only `echo` ran without an approval request (Claude Code treats it as safe). |
10571057
| 2026-11-10 | `copilot` | copilot-sdk-harness | ✅ pass | — | After review round 4 (event correlation to the turn's own user.message, durable state): one turn through LocalSquadBackend completed (5.7 s), and the same with a harness-only Copilot configDirectory in an empty temp dir (3.2 s): sign-in still works there, so state-mcp's isolated config directory is safe. |
1058+
| 2026-11-10 | `claude` | claude-agent-sdk-golden-trace | ✅ pass | `Bash` | Historical smoke run of the original golden-trace test on the real claude-agent-sdk harness (Agent SDK 0.3.317, Anthropic API key), 8.1 s: a real Bash `touch` approval was approved after reconnect and the file was created; the broker recorded cancellation and the fixture's backend events appeared in order. Review found that the assertions did not prove replay continuity, cursor filtering or interruption of an executing sleep. This pass is not evidence for the strengthened trace; a new credentialed live run is pending. |
10581059
| 2026-11-10 | `claude` | plugin-mcp-headless | ✅ pass | `mcp__plugin_squad_squad_state__squad_state_health`, `mcp__squad_state__squad_state_health` | Claude Code 2.1.318, clean environment, `claude -p --output-format stream-json`, no `--mcp-config`, no prior approval. Plugin loaded with `--plugin-dir`: init lists `plugin:squad:squad_state` connected (source `plugin`) and `squad_state_health` returns, at the repo root and from packages/api. The plugin's settings.json `agent: "squad"` made `squad:squad` the main agent with no `--agent`; when the repo also has a projected `.claude/agents/squad.md`, that project agent wins. Control: a project `.mcp.json` server also connected headless without the flag (source `project`) on this version. |
10591060
| 2026-11-10 | `claude` | plugin-mcp-interactive | ✅ pass | — | Claude Code 2.1.318 interactive TUI driven in a pty, clean environment. With the plugin (`--plugin-dir`): after the folder-trust dialog there is no MCP approval prompt, `/mcp` lists `plugin:squad:squad_state` connected (15 tools), and the session starts as `@squad:squad` with no `--agent`. Control, project `.mcp.json` only: after the folder-trust dialog Claude Code shows "New MCP server found in this project: squad_state" with "Continue without using this MCP server" preselected. |
10601061

‎test-fixtures/spec/suite-v0.2/host-feature-matrix.json‎

Lines changed: 13 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -4405,9 +4405,10 @@
44054405
"validated-test:test/harness-contract.test.ts::*",
44064406
"validated-test:test/claude-agent-sdk-harness.test.ts::*",
44074407
"validated-test:test/claude-agent-sdk-harness.test.ts::routes a real tool call through broker approval and honours a deny",
4408-
"validated-manual:lab:claude-agent-sdk-harness@2026-11-10"
4408+
"validated-manual:lab:claude-agent-sdk-harness@2026-11-10",
4409+
"validated-test:test/claude-agent-sdk-harness.test.ts::runs G1–G9 on claude-agent-sdk: approve after reconnect, then cancel"
44094410
],
4410-
"notes": "Live (2026-11-10): one turn through the broker, and a write command approved-or-denied through the broker (denied: not run). Claude Code runs some read-only commands without asking, so those never reach the broker."
4411+
"notes": "Live (2026-11-10): a plain turn and a write denied through the broker (not run). The original golden-trace smoke run approved a real tool call after reconnect, but did not prove replay continuity or prompt harness cancellation. The strengthened trace (exact command and full hashed Bash input validation, explicit 120-second foreground tool timeout, cursor suffix and replay checks, 10-second cancellation deadline) awaits a new credentialed live run and is not yet promotion evidence. Deterministic negatives reject short-timeout/abort-ignoring transport, background execution and implicit timeouts before approval. Claude Code runs some read-only commands without asking, so those never reach the broker."
44114412
},
44124413
"kiro": {
44134414
"capability": "supported",
@@ -5022,6 +5023,16 @@
50225023
"observedTools": [],
50235024
"notes": "After review round 4 (event correlation to the turn's own user.message, durable state): one turn through LocalSquadBackend completed (5.7 s), and the same with a harness-only Copilot configDirectory in an empty temp dir (3.2 s): sign-in still works there, so state-mcp's isolated config directory is safe."
50245025
},
5026+
{
5027+
"host": "claude",
5028+
"scenario": "claude-agent-sdk-golden-trace",
5029+
"date": "2026-11-10",
5030+
"outcome": "pass",
5031+
"observedTools": [
5032+
"Bash"
5033+
],
5034+
"notes": "Historical smoke run of the original golden-trace test on the real claude-agent-sdk harness (Agent SDK 0.3.317, Anthropic API key), 8.1 s: a real Bash `touch` approval was approved after reconnect and the file was created; the broker recorded cancellation and the fixture's backend events appeared in order. Review found that the assertions did not prove replay continuity, cursor filtering or interruption of an executing sleep. This pass is not evidence for the strengthened trace; a new credentialed live run is pending."
5035+
},
50255036
{
50265037
"host": "claude",
50275038
"scenario": "plugin-mcp-headless",

0 commit comments

Comments
 (0)