Support streaming tool output and deduplication

[timvisher-dd]

Closes xenodium#342
Closes xenodium#343

Checklist

• I agree to communicate (PR description and comments) with the author myself (not AI-generated).
• I've reviewed all code in PR myself and will vouch for its quality.
• I've read and followed the Contributing guidelines.
• I've filed a feature request/discussion for a new feature.
• I've added tests where applicable.
• I've run M-x checkdoc and M-x byte-compile-file.

Problem

The two most popular agent ACPs, codex-acp and claude-agent-acp, perform very poorly in agent-shell when tool executions emit a lot of text:

• codex-acp: O(n²) rendering and massive data transfer. A 35k-line bash command takes ~60s and transfers ~890 MB of JSON — a 3,000× amplification of ~280 KB actual output. Each tool_call_update carries the full accumulated output; agent-shell replaces the entire fragment body and reruns markdown-overlays-put on every update.

• claude-agent-acp: output is silently lost. The same command truncates to 241 of 35,001 lines. The user sees raw <persisted-output> XML tags rendered verbatim in the shell buffer.

Cause

agent-shell does not advertise _meta.terminal_output in clientCapabilities during the ACP initialize handshake. Without this capability:

• codex-acp falls back to sending the full accumulated output in every tool_call_update (O(n²) content growth).
• claude-agent-acp sends a single truncated result at completion instead of streaming the full output.

Fix

Advertise _meta.terminal_output during initialize and handle the resulting streaming behavior:

1. Extend acp.el to accept :terminal-capability and :meta-capabilities on acp-make-initialize-request.
2. Pass those capabilities from agent-shell.el during initialize.
3. Handle incremental _meta.terminal_output.data chunks (codex-acp) and batch _meta.terminal_output results (claude-agent-acp) in a new streaming handler with deduplication.
4. Strip <persisted-output> tags and render previews cleanly.

Implementation

New files

agent-shell-meta.el — extractors for ACP _meta payloads:

• agent-shell--meta-lookup — key lookup handling both symbol and string keys in alists.
• agent-shell--meta-find-tool-response — walks any _meta namespace to find a toolResponse value.
• agent-shell--tool-call-meta-response-text — extracts stdout text from _meta.*.toolResponse in its various shapes (string, alist with stdout key, vector of content blocks).
• agent-shell--tool-call-terminal-output-data — extracts _meta.terminal_output.data.

agent-shell-streaming.el — streaming tool call update handler:

• agent-shell--tool-call-normalize-output — strips markdown fences, strips <persisted-output> XML tags (rendering the preview with font-lock-comment-face), and ensures trailing newlines.
• agent-shell--append-tool-call-output — accumulates streamed output in the state's :tool-calls hash under an :accumulated key per tool call ID.
• agent-shell--handle-tool-call-update-streaming — the main handler, replacing the inline tool_call_update block in agent-shell.el. Three branches:
  1. Terminal data (_meta.terminal_output.data): normalize the chunk, accumulate it, and immediately append it to the fragment body for live streaming.
  2. Meta response (_meta.*.toolResponse): normalize and accumulate silently (rendered only on final update to avoid duplication).
  3. Final update (status is "completed" or "failed"): render accumulated output (or fall back to content text), log to transcript, clean up permission dialogs, and apply title/label updates.
• agent-shell--mark-tool-calls-cancelled — marks all in-progress tool calls as cancelled (called from agent-shell-interrupt).

Changes to agent-shell.el

• (require 'agent-shell-streaming) added.
• The ~50-line inline tool_call_update rendering block is replaced by a single call to agent-shell--handle-tool-call-update-streaming. The metadata save (title/description/command/raw-input/diff) remains inline before the handler call.
• The initialize request now passes :terminal-capability t and :meta-capabilities '((terminal_output . t)) to acp-make-initialize-request.
• agent-shell-interrupt calls agent-shell--mark-tool-calls-cancelled after sending the cancel notification.
• shell-maker-define-major-mode call passes 'agent-shell-mode-map (quoted symbol) instead of the bare variable.

Tests

7 new tests in tests/agent-shell-streaming-tests.el:

• agent-shell--tool-call-meta-response-text-test — extracts text from _meta.claudeCode.toolResponse.stdout.
• agent-shell--tool-call-normalize-output-test — strips fences and ensures trailing newline.
• agent-shell--tool-call-normalize-output-persisted-output-test — strips <persisted-output> tags.
• agent-shell--tool-call-update-writes-output-test — verifies accumulated output is written to the fragment body.
• agent-shell--tool-call-meta-response-no-duplication-test — meta response text is rendered once, not duplicated with content.
• agent-shell-initialize-request-meta-capabilities-test — the initialize request includes _meta.terminal_output.
• agent-shell--tool-call-terminal-output-data-streaming-test — codex-style _meta.terminal_output.data chunks are accumulated and rendered incrementally.

Perf measurements

Test: for x in {0..35000}; do printf 'line %d\n' "$x"; done (35,001 lines)

codex-acp

	measure_ms (avg)	content_bytes (avg)	terminal_bytes (avg)
Without terminal caps	~60,000	~900,000,000	0
With terminal caps	~7,500	~3,000	~240,000

~8× faster. Content drops from ~900 MB to ~3 KB.

claude-agent-acp

	measure_ms (avg)	content_bytes	terminal_bytes
Without terminal caps	~22,000	2,321 (truncated to 241 lines)	0
With terminal caps	~23,000	0	2,270

No timing improvement (execution is server-side), but <persisted-output> tags are handled cleanly.

Prerequisite: acp.el changes

acp.el needs to accept :terminal-capability and :meta-capabilities keyword arguments on acp-make-initialize-request. See xenodium/acp.el#15.