Skip to content

fix(supervisor): stop terminate() racing watchChild's Cmd.Wait()#62

Merged
nhuelstng merged 2 commits into
mainfrom
fix/supervisor-concurrent-wait-deadlock
Jul 13, 2026
Merged

fix(supervisor): stop terminate() racing watchChild's Cmd.Wait()#62
nhuelstng merged 2 commits into
mainfrom
fix/supervisor-concurrent-wait-deadlock

Conversation

@nhuelstng

Copy link
Copy Markdown
Contributor

What

Fixes a data race between terminate() and watchChild()'s reaper goroutine, both of which called os/exec.Cmd.Wait() concurrently on the same sidecar process.

Why

This is the root cause of the agent did not exit within 5m0s E2E hangs investigated in #60/#61 — every matrix job registers at least one sidecar (echo-rest/self-audit), so every harness (including claude-code) is equally exposed, which is why #61's "shared GLM-backend contention" theory stopped holding up once claude-code hung too in a later run.

watchChild() (added in #38) spawns a goroutine that calls r.Cmd.Wait() on a sidecar as soon as it starts, so process.exit can be audited at real termination time. But terminate() (called by StopSidecar/ShutdownAll at teardown) also spawns a goroutine calling cmd.Wait() on the same *exec.Cmd. os/exec.Cmd.Wait must only ever have one caller in flight — calling it concurrently races on shared internal state, and the loser can block forever reading a channel the other caller already drained, even though the child process has actually exited. Once that happens, ShutdownAll's WaitGroup.Wait() never returns, so the parent omac start process (which the e2e test's cmd.Run() is blocked on) never exits either.

How

Running now carries a waitDone channel that watchChild closes after it reaps the child. terminate() waits on that channel instead of calling Cmd.Wait() itself whenever a reaper is already running (any child reachable via s.children — i.e. StopSidecar and ShutdownAll). The one call site that runs before watchChild starts (a health-check failure during sidecar startup) still spawns its own Wait(), since no reaper exists yet there.

Verification

  • Goroutine dump from a real hang: reproduced locally (register echo-rest, omac start opencode --no-sandbox in a loop against a fake fast-exiting harness) and captured a SIGQUIT dump showing the main goroutine wedged in ShutdownAll's WaitGroup.Wait()terminate()Cmd.Wait(), forever.
  • Deterministic repro, before/after: same loop hung on every iteration after the first, 100% reproducible, before this fix; 10/10 clean runs (~0.5s each) after, both under --no-sandbox and under the real bubblewrap sandbox.
  • go test -race: added TestShutdownAllReapsLongRunningChildWithoutHanging — confirmed it fails with WARNING: DATA RACE between terminate.func1 and watchChild.func1 on the pre-fix code (reverted via patch to check), and passes clean post-fix.
  • go build ./..., go vet ./..., gofmt -l, full go test ./... (non-e2e) — all clean.
  • Full internal/supervisor suite passes under -race.

Relationship to #60 / #61

The audit-trail commit (#38) added a watchChild reaper goroutine that
calls Cmd.Wait() on a sidecar as soon as it starts, so process.exit can
be audited at actual termination time. But terminate() (used by
StopSidecar/ShutdownAll) also spawned its own goroutine calling
Cmd.Wait() on the same *exec.Cmd. os/exec.Cmd.Wait must only ever have
one caller in flight; concurrent callers race on shared internal state,
and the loser can block forever reading a channel the other caller has
already drained — even after the child has actually exited.

This is the root cause of "agent did not exit within 5m0s" e2e hangs
observed across every harness (opencode/codex/copilot/claude-code all
register sidecars, so all are equally exposed) — confirmed with:
  - a goroutine dump from a hung process, showing the main goroutine
    stuck in ShutdownAll's WaitGroup.Wait() -> terminate() -> Cmd.Wait(),
  - a 10-iteration local repro (register echo-rest, `omac start` in a
    loop) that hung deterministically on every run after the first
    before this fix, and 0/10 after,
  - `go test -race` on a new regression test, which flags the exact
    data race between terminate.func1 and watchChild.func1 on the same
    Cmd before this fix, and passes clean after.

Fix: Running now carries a waitDone channel that watchChild closes
after it reaps the child. terminate() waits on that channel instead of
calling Cmd.Wait() itself whenever a reaper is already running (i.e.
for any child reachable via s.children). The one call site that runs
before watchChild starts (a health-check failure during startup) still
spawns its own Wait(), since no reaper exists yet there.

This supersedes the speculative "reduce E2E concurrency" mitigation in
a sibling PR — that PR's theory (shared GLM-backend contention) didn't
hold up: claude-code, which uses a separate backend, hung with the same
signature in a later run. This fix addresses the actual, reproducible
cause and isn't backend- or harness-specific.

Signed-off-by: Niclas Hülsmann <niclas.huelsmann@tngtech.com>
allHarnesses() has excluded codex on darwin since cba169f (its Rust
HTTP client is incompatible with the macOS Seatbelt sandbox — issue
#48), but TestAllHarnessesReturnsFour and TestHarnessByName still
hardcoded an expectation of exactly 4 harnesses including codex. This
has failed deterministically on every macOS E2E job since that commit
landed, visible in CI as harnesses_test.go:12 "expected 4 harnesses,
got 3" and harnesses_test.go:20 "harnessByName(\"codex\") not found".

Derive the expected harness set from GOOS the same way allHarnesses()
does, so the tests track that exclusion instead of duplicating a stale
assumption.

Signed-off-by: Niclas Hülsmann <niclas.huelsmann@tngtech.com>
@nhuelstng nhuelstng merged commit 95e66f7 into main Jul 13, 2026
7 of 11 checks passed
@nhuelstng nhuelstng deleted the fix/supervisor-concurrent-wait-deadlock branch July 13, 2026 11:18
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant