You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Commit 9dd3700
Browse filesBrowse the repository at this point in the historyBrowse files
authored
test(backend): run the golden trace live on the claude-agent-sdk harness (#2441) (#2452)
* test(backend): run the golden trace live on the claude-agent-sdk harness
golden-trace.json (decoupling §14.4, G1–G9) previously ran only against a
fake harness. A live variant now runs it on the real Claude Agent SDK
behind SQUAD_REAL_CLI_TESTS=1 and an Anthropic API key: a real Bash
approval survives a client disconnect and cursor reconnect, the broker
approves it and the command runs (the first live approve), then a cancel
ends the turn cancelled, with the fixture's backend events in order.
Passed 2026-11-10 (Agent SDK 0.3.317, 8.1 s). Recorded in the host
feature matrix and decoupling spec §12. Env-gated in CI by the existing
reconciliation rule.
Refs #2441
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* test(backend): tighten live golden trace review assertions
Part of #2441. Address existing feedback on PR #2452: verify replay continuity and cursor suffixes, approve only the exact foreground command, require prompt harness cancellation, and cancel/settle on failure. Qualify historical smoke evidence pending a strengthened live rerun.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: c8c234f1-ee3b-49e1-9406-11d4a2238c23
* test(backend): validate golden Bash execution options
Part of #2441. Complete PR #2452's cancellation finding with test-owned full-input interception, explicit long foreground timeout and hash binding before approval, and deterministic short-timeout/background/default negatives plus an abort-honouring control. Keep strengthened live promotion evidence pending.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: c8c234f1-ee3b-49e1-9406-11d4a2238c23
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: c8c234f1-ee3b-49e1-9406-11d4a2238c23
| `copilot` | ✅ supported | 🖥️ validated-live | Opt-in tools over the same MCP server; backend semantics are host-independent. The `copilot-sdk` harness (`CopilotSdkHarness`) executes turns on the Copilot SDK; opt-in on the bridge via `SQUAD_BACKEND_HARNESSES=copilot-sdk`. _Live: plain turns through the broker on the real Copilot backend (2026-11-09; again 2026-11-10 after event correlation, including with a harness-only config directory). Approvals are covered by the shared contract, not yet live._ | `validated-test:test/squad-20-real-clients-stdio.e2e.test.ts::completes the MCP handshake and lists squad_state_* and all squad_backend_* tools when opted in`<br>`validated-test:test/squad-20-all-scenarios.e2e.test.ts::Scenario 4: DurableHttpSquadBackend full HTTP lifecycle`<br>`validated-test:test/squad-20-all-scenarios.e2e.test.ts::Full non-bypass journey: host adapter → broker → harness → approval → restart/reconnect`<br>`validated-test:test/harness-contract.test.ts::*`<br>`validated-test:test/copilot-sdk-harness.test.ts::completes a turn through the broker`<br>`validated-test:test/copilot-sdk-harness.test.ts::completes a turn with a harness-only Copilot config directory (login still works)`<br>`validated-manual:lab:copilot-sdk-harness@2026-11-10` |
975
-
| `claude` | ✅ supported | 🖥️ validated-live | Opt-in tools over the same MCP server; backend semantics are host-independent. The `claude-agent-sdk` harness (`ClaudeAgentSdkHarness`) executes turns on the Claude Agent SDK when the optional peer is installed. _Live (2026-11-10): one turn through the broker, and a write command approved-or-denied through the broker (denied: not run). Claude Code runs some read-only commands without asking, so those never reach the broker._ | `validated-test:test/squad-20-real-clients-stdio.e2e.test.ts::completes the MCP handshake and lists squad_state_* and all squad_backend_* tools when opted in`<br>`validated-test:test/squad-20-all-scenarios.e2e.test.ts::Scenario 4: DurableHttpSquadBackend full HTTP lifecycle`<br>`validated-test:test/squad-20-all-scenarios.e2e.test.ts::Full non-bypass journey: host adapter → broker → harness → approval → restart/reconnect`<br>`validated-test:test/harness-contract.test.ts::*`<br>`validated-test:test/claude-agent-sdk-harness.test.ts::*`<br>`validated-test:test/claude-agent-sdk-harness.test.ts::routes a real tool call through broker approval and honours a deny`<br>`validated-manual:lab:claude-agent-sdk-harness@2026-11-10` |
975
+
| `claude` | ✅ supported | 🖥️ validated-live | Opt-in tools over the same MCP server; backend semantics are host-independent. The `claude-agent-sdk` harness (`ClaudeAgentSdkHarness`) executes turns on the Claude Agent SDK when the optional peer is installed. _Live (2026-11-10): a plain turn and a write denied through the broker (not run). The original golden-trace smoke run approved a real tool call after reconnect, but did not prove replay continuity or prompt harness cancellation. The strengthened trace (exact command and full hashed Bash input validation, explicit 120-second foreground tool timeout, cursor suffix and replay checks, 10-second cancellation deadline) awaits a new credentialed live run and is not yet promotion evidence. Deterministic negatives reject short-timeout/abort-ignoring transport, background execution and implicit timeouts before approval. Claude Code runs some read-only commands without asking, so those never reach the broker._ | `validated-test:test/squad-20-real-clients-stdio.e2e.test.ts::completes the MCP handshake and lists squad_state_* and all squad_backend_* tools when opted in`<br>`validated-test:test/squad-20-all-scenarios.e2e.test.ts::Scenario 4: DurableHttpSquadBackend full HTTP lifecycle`<br>`validated-test:test/squad-20-all-scenarios.e2e.test.ts::Full non-bypass journey: host adapter → broker → harness → approval → restart/reconnect`<br>`validated-test:test/harness-contract.test.ts::*`<br>`validated-test:test/claude-agent-sdk-harness.test.ts::*`<br>`validated-test:test/claude-agent-sdk-harness.test.ts::routes a real tool call through broker approval and honours a deny`<br>`validated-manual:lab:claude-agent-sdk-harness@2026-11-10`<br>`validated-test:test/claude-agent-sdk-harness.test.ts::runs G1–G9 on claude-agent-sdk: approve after reconnect, then cancel` |
976
976
|`kiro`| ✅ supported | 🧪 validated-test | Opt-in tools over the same MCP server; backend semantics are host-independent. |`validated-test:test/squad-20-real-clients-stdio.e2e.test.ts::completes the MCP handshake and lists squad_state_* and all squad_backend_* tools when opted in`<br>`validated-test:test/squad-20-all-scenarios.e2e.test.ts::Scenario 4: DurableHttpSquadBackend full HTTP lifecycle`<br>`validated-test:test/squad-20-all-scenarios.e2e.test.ts::Full non-bypass journey: host adapter → broker → harness → approval → restart/reconnect`|
977
977
|`opencode`| ✅ supported | 🧪 validated-test | Opt-in tools over the same MCP server; backend semantics are host-independent. |`validated-test:test/squad-20-real-clients-stdio.e2e.test.ts::completes the MCP handshake and lists squad_state_* and all squad_backend_* tools when opted in`<br>`validated-test:test/squad-20-all-scenarios.e2e.test.ts::Scenario 4: DurableHttpSquadBackend full HTTP lifecycle`<br>`validated-test:test/squad-20-all-scenarios.e2e.test.ts::Full non-bypass journey: host adapter → broker → harness → approval → restart/reconnect`|
978
978
|`cursor`| ✅ supported | 🧪 validated-test | Opt-in tools over the same MCP server; backend semantics are host-independent. |`validated-test:test/squad-20-real-clients-stdio.e2e.test.ts::completes the MCP handshake and lists squad_state_* and all squad_backend_* tools when opted in`<br>`validated-test:test/squad-20-all-scenarios.e2e.test.ts::Scenario 4: DurableHttpSquadBackend full HTTP lifecycle`<br>`validated-test:test/squad-20-all-scenarios.e2e.test.ts::Full non-bypass journey: host adapter → broker → harness → approval → restart/reconnect`|
@@ -1055,6 +1055,7 @@ Headless real-CLI runs behind the `validated-manual:lab:<scenario>` evidence ent
1055
1055
| 2026-11-09 |`copilot`| copilot-sdk-harness | ✅ pass | — | copilot-sdk HarnessAdapter (@github/copilot-sdk 1.0.19, signed-in Copilot user) completed one turn through LocalSquadBackend: "Reply with exactly the word: pong" → completed, transcript ends in pong (4.4 s). Plain text only; no tool approval in this run. |
1056
1056
| 2026-11-10 |`claude`| claude-agent-sdk-harness | ✅ pass |`Bash`| claude-agent-sdk HarnessAdapter (@anthropic-ai/claude-agent-sdk 0.3.317, Anthropic API key) through LocalSquadBackend: a plain turn completed (6.7 s); a Bash `touch` reached the broker as an approval, was denied, ended the turn cancelled, and the file was not created (6.1 s). A read-only `echo` ran without an approval request (Claude Code treats it as safe). |
1057
1057
| 2026-11-10 |`copilot`| copilot-sdk-harness | ✅ pass | — | After review round 4 (event correlation to the turn's own user.message, durable state): one turn through LocalSquadBackend completed (5.7 s), and the same with a harness-only Copilot configDirectory in an empty temp dir (3.2 s): sign-in still works there, so state-mcp's isolated config directory is safe. |
1058
+
| 2026-11-10 |`claude`| claude-agent-sdk-golden-trace | ✅ pass |`Bash`| Historical smoke run of the original golden-trace test on the real claude-agent-sdk harness (Agent SDK 0.3.317, Anthropic API key), 8.1 s: a real Bash `touch` approval was approved after reconnect and the file was created; the broker recorded cancellation and the fixture's backend events appeared in order. Review found that the assertions did not prove replay continuity, cursor filtering or interruption of an executing sleep. This pass is not evidence for the strengthened trace; a new credentialed live run is pending. |
1058
1059
| 2026-11-10 |`claude`| plugin-mcp-headless | ✅ pass |`mcp__plugin_squad_squad_state__squad_state_health`, `mcp__squad_state__squad_state_health`| Claude Code 2.1.318, clean environment, `claude -p --output-format stream-json`, no `--mcp-config`, no prior approval. Plugin loaded with `--plugin-dir`: init lists `plugin:squad:squad_state` connected (source `plugin`) and `squad_state_health` returns, at the repo root and from packages/api. The plugin's settings.json `agent: "squad"` made `squad:squad` the main agent with no `--agent`; when the repo also has a projected `.claude/agents/squad.md`, that project agent wins. Control: a project `.mcp.json` server also connected headless without the flag (source `project`) on this version. |
1059
1060
| 2026-11-10 |`claude`| plugin-mcp-interactive | ✅ pass | — | Claude Code 2.1.318 interactive TUI driven in a pty, clean environment. With the plugin (`--plugin-dir`): after the folder-trust dialog there is no MCP approval prompt, `/mcp` lists `plugin:squad:squad_state` connected (15 tools), and the session starts as `@squad:squad` with no `--agent`. Control, project `.mcp.json` only: after the folder-trust dialog Claude Code shows "New MCP server found in this project: squad_state" with "Continue without using this MCP server" preselected. |
"validated-test:test/claude-agent-sdk-harness.test.ts::runs G1–G9 on claude-agent-sdk: approve after reconnect, then cancel"
4409
4410
],
4410
-
"notes": "Live (2026-11-10): one turn through the broker, and a write command approved-or-denied through the broker (denied: not run). Claude Code runs some read-only commands without asking, so those never reach the broker."
4411
+
"notes": "Live (2026-11-10): a plain turn and a write denied through the broker (not run). The original golden-trace smoke run approved a real tool call after reconnect, but did not prove replay continuity or prompt harness cancellation. The strengthened trace (exact command and full hashed Bash input validation, explicit 120-second foreground tool timeout, cursor suffix and replay checks, 10-second cancellation deadline) awaits a new credentialed live run and is not yet promotion evidence. Deterministic negatives reject short-timeout/abort-ignoring transport, background execution and implicit timeouts before approval. Claude Code runs some read-only commands without asking, so those never reach the broker."
4411
4412
},
4412
4413
"kiro": {
4413
4414
"capability": "supported",
@@ -5022,6 +5023,16 @@
5022
5023
"observedTools": [],
5023
5024
"notes": "After review round 4 (event correlation to the turn's own user.message, durable state): one turn through LocalSquadBackend completed (5.7 s), and the same with a harness-only Copilot configDirectory in an empty temp dir (3.2 s): sign-in still works there, so state-mcp's isolated config directory is safe."
5024
5025
},
5026
+
{
5027
+
"host": "claude",
5028
+
"scenario": "claude-agent-sdk-golden-trace",
5029
+
"date": "2026-11-10",
5030
+
"outcome": "pass",
5031
+
"observedTools": [
5032
+
"Bash"
5033
+
],
5034
+
"notes": "Historical smoke run of the original golden-trace test on the real claude-agent-sdk harness (Agent SDK 0.3.317, Anthropic API key), 8.1 s: a real Bash `touch` approval was approved after reconnect and the file was created; the broker recorded cancellation and the fixture's backend events appeared in order. Review found that the assertions did not prove replay continuity, cursor filtering or interruption of an executing sleep. This pass is not evidence for the strengthened trace; a new credentialed live run is pending."
0 commit comments