fix(agent-runner): reconcile Claude tool allowlist with pinned CLI, add drift guard#2982
Closed
gabi-simons wants to merge 1 commit into
Closed
fix(agent-runner): reconcile Claude tool allowlist with pinned CLI, add drift guard#2982gabi-simons wants to merge 1 commit into
gabi-simons wants to merge 1 commit into
Conversation
ad2bc65 to
7ce25de
Compare
…dd drift guard TOOL_ALLOWLIST named five tools that don't exist on claude-code 2.1.197 (Task was renamed Agent upstream; TodoWrite, TeamCreate, TeamDelete, and ToolSearch are gone). Remove the phantoms and correct the comment (and the stale claim in docs/agent-runner-details.md): allowedTools is a permission auto-approve list, not an availability filter — wire captures show no allowlist effect on the stable tool surface, and this runner's bypassPermissions moots its permission role. No wire-visible change. Wire measurement on this pin shows PER-QUERY nondeterminism in the offered surface: conditional tools (Glob/Grep) flicker, and disallowedTools stripping of flag-gated tools is best-effort. The fixture therefore captures N interleaved rounds per mode and asserts equality on the STABLE cores, with the flicker set recorded separately (variantTools) so regeneration cannot flake and a real allowlist- semantics change fails deterministically. A disallow probe is recorded as a diagnostic; the deterministic tool block remains the runner's PreToolUse hook. claude.tools.test.ts fails on any future claude-code CLI or Agent SDK bump until the fixture is regenerated with dump-sdk-tools.ts and the lists re-verified. The SDK dep is pinned exactly (0.3.197, was caret) so drift can only enter deliberately. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
7ce25de to
d548980
Compare
This was referenced Jul 13, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
TOOL_ALLOWLISTin the agent runner named five tools that don't exist on the pinned claude-code CLI (2.1.197):Task(renamedAgentupstream),TodoWrite,TeamCreate,TeamDelete, andToolSearch. The comment above it also claimed the SDK'sallowedToolsfilters out unlisted MCP namespaces — which is not how the SDK behaves.This PR fixes the list to the 14 names that actually exist, rewrites the comment to match verified behavior, and adds a drift guard so the list can't silently rot again when the CLI or SDK pin moves.
Verified behavior (wire-captured)
allowedToolsis a permission auto-approve list, not an availability filter: interleaved wire captures (production allowlist vs no allowlist at all) show no allowlist effect on the stable tool surface, and this runner'sbypassPermissionsmoots its permission role.allowedToolssemantics fails tests instead of silently changing agent behavior.disallowedToolsstripping of flag-gated tools is best-effort (each single query is internally coherent). Because the variance is per-query, no single capture pair can isolate the allowlist — so the fixture takes N interleaved rounds per mode and asserts equality on the stable cores (tools present in every round of a mode), with the flicker set recorded separately asvariantTools. Regeneration therefore cannot flake on the variance, while a real allowlist-semantics change diverges the cores in every round and fails deterministically. A disallow probe is recorded as a diagnostic; the deterministic tool block in this runner is the PreToolUse hook, unchanged here.Drift guard
sdk-tools-baseline.json— a wire-captured fixture (tools/toolsBare= stable cores over interleaved rounds,variantTools= flicker set,toolsDisallowProbediagnostic) of every tool the pinned CLI offers under our configuration. Captured with zero API cost:dump-sdk-tools.tsruns inside the agent image withANTHROPIC_BASE_URLpointed at an in-process 401 stub; the full tools array rides on the first request, which is captured before the run dies on the auth error. The regeneration command is documented in the script header.claude.tools.test.ts— asserts the stable cores are identical (the no-surface-effect invariant), every allowlist/disallow entry names a real tool (membership checked against stable ∪ variant) (with a documentedKNOWN_ABSENTset for disallow entries kept as drift insurance), and that the fixture's versions match both thecli-tools.jsonCLI pin and the installed Agent SDK — so any future pin bump fails tests until the fixture is regenerated and the lists re-verified.@anthropic-ai/claude-agent-sdkis pinned exactly (0.3.197, was caret) so an SDK bump can only enter deliberately, together with a fixture regeneration.Testing
cd container/agent-runner && bun test— 123 pass (6 drift-guard tests) — andbun run typecheckclean.🤖 Generated with Claude Code