fix(agent-runner): stall watchdog to recover from hung in-flight tools#3019
Open
Shufel83 wants to merge 1 commit into
Open
fix(agent-runner): stall watchdog to recover from hung in-flight tools#3019Shufel83 wants to merge 1 commit into
Shufel83 wants to merge 1 commit into
Conversation
A single tool call that never returns (a hung MCP/network request, a wedged browse, a script with no timeout) parks the SDK in its message loop: no `activity` event flows, the heartbeat freezes, and the container sits dead until the host's 30-min absolute ceiling kills it — after which the reset messages re-fire and it re-wedges. Add a container-side watchdog in processQuery that reads the in-flight tool from container_state (recorded by the PreToolUse hook) and, if a tool has been running past its budget (default 10 min; Bash honours its own declared timeout), logs which tool stalled, aborts the query, and exits for a host respawn. Fires only on an over-budget in-flight tool, so an idle open query between turns is never touched and non-tool stalls still fall back to the host ceiling. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This was referenced Jul 12, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
On a busy agent group we saw the container repeatedly killed by the host's absolute ceiling — every kill logged with
heartbeatAgeMs ≈ 1,800,000(a full 30 min) againstceilingMs=1800000. That means a genuine 30-minute stretch of zero SDK activity, not accumulated short gaps.Root cause: a single tool call that never returns — a hung MCP/network request, a wedged browse, a script with no timeout — parks the SDK in its
for awaitmessage loop. Noactivityevent flows, so the heartbeat (only touched on provider events during a query) freezes, and the container sits dead until the 30-min ceiling finally kills it. The reset messages then re-fire on respawn and it re-wedges. Across our logs this accounted for well over 100 kills, concentrated on the busiest group.Fix
Add a container-side stall watchdog in
processQuery. ThePreToolUsehook already records the in-flight tool + start time incontainer_state(cleared byPostToolUse); a newgetContainerToolInFlight()reader exposes it. A 30s-tick watchdog checks whether a tool has been in flight past its budget:TOOL_STALL_MS= 10 min.max(TOOL_STALL_MS, declaredTimeoutMs + grace).On stall it logs which tool stalled (the host kill logs never captured that), calls
query.abort(), and — sinceabort()only unblocks the SDK when it's parked awaiting input, not on a genuinely hung tool — hard-exits after a short grace so the host respawns a fresh container. Recovery drops from ~30 min to ~10.Scope / safety
container_state, not the heartbeat, so it works even when other activity keeps the heartbeat artificially fresh.Testing
Typechecks. Deployed live on our install (agent-runner source is RO-mounted, no image rebuild). Note: the container test suite (
bun test) wasn't run in this environment — no Bun on the host outside the image.