Skip to content

fix(agent-runner): reconcile Claude tool allowlist with pinned CLI, add drift guard#2982

Closed
gabi-simons wants to merge 1 commit into
mainfrom
fix/claude-allowlist-reconciliation
Closed

fix(agent-runner): reconcile Claude tool allowlist with pinned CLI, add drift guard#2982
gabi-simons wants to merge 1 commit into
mainfrom
fix/claude-allowlist-reconciliation

Conversation

@gabi-simons

@gabi-simons gabi-simons commented Jul 8, 2026

Copy link
Copy Markdown
Collaborator

What

TOOL_ALLOWLIST in the agent runner named five tools that don't exist on the pinned claude-code CLI (2.1.197): Task (renamed Agent upstream), TodoWrite, TeamCreate, TeamDelete, and ToolSearch. The comment above it also claimed the SDK's allowedTools filters out unlisted MCP namespaces — which is not how the SDK behaves.

This PR fixes the list to the 14 names that actually exist, rewrites the comment to match verified behavior, and adds a drift guard so the list can't silently rot again when the CLI or SDK pin moves.

Verified behavior (wire-captured)

  • allowedTools is a permission auto-approve list, not an availability filter: interleaved wire captures (production allowlist vs no allowlist at all) show no allowlist effect on the stable tool surface, and this runner's bypassPermissions moots its permission role.
  • The list is kept as the accurate pre-approval set for a hypothetical non-bypass mode; the drift guard asserts the no-surface-effect invariant so a future CLI/SDK version that changes allowedTools semantics fails tests instead of silently changing agent behavior.
  • Wire measurement also surfaced per-query nondeterminism in the offered surface on this pin: conditional tools (Glob/Grep) flicker across query invocations, and disallowedTools stripping of flag-gated tools is best-effort (each single query is internally coherent). Because the variance is per-query, no single capture pair can isolate the allowlist — so the fixture takes N interleaved rounds per mode and asserts equality on the stable cores (tools present in every round of a mode), with the flicker set recorded separately as variantTools. Regeneration therefore cannot flake on the variance, while a real allowlist-semantics change diverges the cores in every round and fails deterministically. A disallow probe is recorded as a diagnostic; the deterministic tool block in this runner is the PreToolUse hook, unchanged here.
  • Net effect of this PR on the wire: none — every removed name was a phantom.

Drift guard

  • sdk-tools-baseline.json — a wire-captured fixture (tools/toolsBare = stable cores over interleaved rounds, variantTools = flicker set, toolsDisallowProbe diagnostic) of every tool the pinned CLI offers under our configuration. Captured with zero API cost: dump-sdk-tools.ts runs inside the agent image with ANTHROPIC_BASE_URL pointed at an in-process 401 stub; the full tools array rides on the first request, which is captured before the run dies on the auth error. The regeneration command is documented in the script header.
  • claude.tools.test.ts — asserts the stable cores are identical (the no-surface-effect invariant), every allowlist/disallow entry names a real tool (membership checked against stable ∪ variant) (with a documented KNOWN_ABSENT set for disallow entries kept as drift insurance), and that the fixture's versions match both the cli-tools.json CLI pin and the installed Agent SDK — so any future pin bump fails tests until the fixture is regenerated and the lists re-verified.
  • @anthropic-ai/claude-agent-sdk is pinned exactly (0.3.197, was caret) so an SDK bump can only enter deliberately, together with a fixture regeneration.

Testing

  • cd container/agent-runner && bun test — 123 pass (6 drift-guard tests) — and bun run typecheck clean.
  • Wire probe before/after confirms an identical tool surface.

🤖 Generated with Claude Code

…dd drift guard

TOOL_ALLOWLIST named five tools that don't exist on claude-code 2.1.197
(Task was renamed Agent upstream; TodoWrite, TeamCreate, TeamDelete, and
ToolSearch are gone). Remove the phantoms and correct the comment (and
the stale claim in docs/agent-runner-details.md): allowedTools is a
permission auto-approve list, not an availability filter — wire captures
show no allowlist effect on the stable tool surface, and this runner's
bypassPermissions moots its permission role. No wire-visible change.

Wire measurement on this pin shows PER-QUERY nondeterminism in the
offered surface: conditional tools (Glob/Grep) flicker, and
disallowedTools stripping of flag-gated tools is best-effort. The
fixture therefore captures N interleaved rounds per mode and asserts
equality on the STABLE cores, with the flicker set recorded separately
(variantTools) so regeneration cannot flake and a real allowlist-
semantics change fails deterministically. A disallow probe is recorded
as a diagnostic; the deterministic tool block remains the runner's
PreToolUse hook.

claude.tools.test.ts fails on any future claude-code CLI or Agent SDK
bump until the fixture is regenerated with dump-sdk-tools.ts and the
lists re-verified. The SDK dep is pinned exactly (0.3.197, was caret)
so drift can only enter deliberately.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

core-team PR opened by a core team member

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants