Skip to content

fix(agent-runner): stall watchdog to recover from hung in-flight tools#3019

Open
Shufel83 wants to merge 1 commit into
nanocoai:mainfrom
Shufel83:fix/stall-watchdog
Open

fix(agent-runner): stall watchdog to recover from hung in-flight tools#3019
Shufel83 wants to merge 1 commit into
nanocoai:mainfrom
Shufel83:fix/stall-watchdog

Conversation

@Shufel83

Copy link
Copy Markdown

Problem

On a busy agent group we saw the container repeatedly killed by the host's absolute ceiling — every kill logged with heartbeatAgeMs ≈ 1,800,000 (a full 30 min) against ceilingMs=1800000. That means a genuine 30-minute stretch of zero SDK activity, not accumulated short gaps.

Root cause: a single tool call that never returns — a hung MCP/network request, a wedged browse, a script with no timeout — parks the SDK in its for await message loop. No activity event flows, so the heartbeat (only touched on provider events during a query) freezes, and the container sits dead until the 30-min ceiling finally kills it. The reset messages then re-fire on respawn and it re-wedges. Across our logs this accounted for well over 100 kills, concentrated on the busiest group.

Fix

Add a container-side stall watchdog in processQuery. The PreToolUse hook already records the in-flight tool + start time in container_state (cleared by PostToolUse); a new getContainerToolInFlight() reader exposes it. A 30s-tick watchdog checks whether a tool has been in flight past its budget:

  • Default budget TOOL_STALL_MS = 10 min.
  • Bash honours its own declared timeout the same way the host ceiling does: max(TOOL_STALL_MS, declaredTimeoutMs + grace).

On stall it logs which tool stalled (the host kill logs never captured that), calls query.abort(), and — since abort() only unblocks the SDK when it's parked awaiting input, not on a genuinely hung tool — hard-exits after a short grace so the host respawns a fresh container. Recovery drops from ~30 min to ~10.

Scope / safety

  • Fires only on an over-budget in-flight tool, so an idle open query between turns is never touched — no regression for quiet agents.
  • A rare non-tool stall (a pure SDK/model hang with no tool in flight) still falls back to the existing host ceiling, unchanged.
  • Reads container_state, not the heartbeat, so it works even when other activity keeps the heartbeat artificially fresh.

Testing

Typechecks. Deployed live on our install (agent-runner source is RO-mounted, no image rebuild). Note: the container test suite (bun test) wasn't run in this environment — no Bun on the host outside the image.

A single tool call that never returns (a hung MCP/network request, a
wedged browse, a script with no timeout) parks the SDK in its message
loop: no `activity` event flows, the heartbeat freezes, and the container
sits dead until the host's 30-min absolute ceiling kills it — after which
the reset messages re-fire and it re-wedges.

Add a container-side watchdog in processQuery that reads the in-flight
tool from container_state (recorded by the PreToolUse hook) and, if a tool
has been running past its budget (default 10 min; Bash honours its own
declared timeout), logs which tool stalled, aborts the query, and exits
for a host respawn. Fires only on an over-budget in-flight tool, so an
idle open query between turns is never touched and non-tool stalls still
fall back to the host ceiling.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant