[Bug]: statusLine/title refresh runner leaks orphaned subprocesses on kill timeout (`TerminateProcess: Access is denied`) — accumulates into `OutOfMemoryException` over sustained use

[RegreDanger]

Antigravity CLI Version

1.1.8

Environment Context

• OS: Windows 11 Pro, build 10.0.22621
• Shell: Windows PowerShell 5.1 (powershell.exe), invoked via a custom statusLine/title command in settings.json
• Terminal Emulator: Windows Terminal
• Runtime Version: PowerShell 5.1.22621.4249
• Hardware Details: 15.88GB RAM total (relevant below — the eventual crash is not caused by physical memory exhaustion)

Steps to Reproduce

• Configure a custom statusLine and/or title command in settings.json pointing to an external script (PowerShell in this case, but the mechanism is independent of the script's language):

  {
    "statusLine": {
      "command": "powershell.exe -NoProfile -ExecutionPolicy Bypass -File C:/Users/<username>/.gemini/antigravity-cli/statusline.ps1",
      "enabled": true
    },
    "title": {
      "command": "powershell.exe -NoProfile -ExecutionPolicy Bypass -File C:/Users/<username>/.gemini/antigravity-cli/title.ps1",
      "enabled": true
    }
  }

• Use agy normally for an extended period (hours). No special action is needed to trigger it — it happens as a background side effect of normal refresh ticks.

• Periodically, a given tick's subprocess invocation runs long enough that the runner decides to kill it (timeout). Check the CLI log at that point (log/cli-*.log).

Expected Behavior

When a statusLine/title subprocess invocation times out, the runner should successfully terminate it and move on cleanly on the next tick, leaving no residual process behind.

Actual Behavior / Error Screenshots

The kill attempt itself fails, and the subprocess is left running indefinitely instead of being cleaned up. The CLI log shows this exact pattern repeating:

statusline: command failed: exit status 1 (stderr: ) (failure 1/30)
statusline: command failed: exec: canceling Cmd: TerminateProcess: Access is denied. (stderr: ) (failure 1/30)

Visibly, this surfaces in the TUI as intermittent:

⚠ Statusline Error
  ⎿  ⚠ Statusline error:
     command failed: exit status 1 (stderr: )
     Full logs at: C:\Users\<username>\.gemini\antigravity-cli\log\cli-....log

Because the failed kill leaves the process alive, this accumulates over time rather than self-correcting. In our case, we found 565 orphaned powershell.exe processes via Task Manager / Get-Process, the oldest dated ~50 hours prior to discovery — all idle (0% CPU, ~1MB working set each, Responding: True), confirmed via Win32_Process.CommandLine (CIM/WMI) to all be invocations of the configured statusLine/title script. The sheer process/handle count (3,790 kernel handles from these alone) eventually degraded new-process creation system-wide enough to throw a genuine System.OutOfMemoryException when the runner tried to spawn the next refresh tick — this is kernel handle/session quota exhaustion, not physical RAM exhaustion (5.48GB remained free at the time).

Note: this reproduces identically with a compiled native binary (e.g. a Go .exe) in place of the PowerShell script instead of powershell.exe -File ... — confirms the bug is in the runner's own process-kill path (exec: canceling Cmd: TerminateProcess), not specific to PowerShell or any particular scripting language.

Logs & Configurations

Relevant excerpt from %APPDATA%\antigravity-cli\log\cli-*.log (timestamps/PIDs are illustrative, pattern repeats verbatim across many separate occurrences over the 50-hour window):

I<timestamp> statusline_runner.go:258] statusline: command failed: exit status 1 (stderr: ) (failure 1/30)
I<timestamp> statusline_runner.go:258] statusline: command failed: exec: canceling Cmd: TerminateProcess: Access is denied. (stderr: ) (failure 1/30)
I<timestamp> statusline_runner.go:258] statusline: command failed: exit status 1 (stderr: ) (failure 1/30)

settings.json (statusLine/title block, path sanitized):

{
  "statusLine": {
    "type": "command",
    "command": "powershell.exe -NoProfile -ExecutionPolicy Bypass -File C:/Users/<username>/.gemini/antigravity-cli/statusline.ps1",
    "enabled": true
  },
  "title": {
    "type": "command",
    "command": "powershell.exe -NoProfile -ExecutionPolicy Bypass -File C:/Users/<username>/.gemini/antigravity-cli/title.ps1",
    "enabled": true
  }
}

Verification commands used to confirm the leak (Windows PowerShell):

Get-CimInstance Win32_Process -Filter "Name='powershell.exe'" |
    Select-Object ProcessId, CreationDate, CommandLine

Happy to provide the full process dump, further log excerpts, or test a proposed fix — this was reproduced consistently over a multi-day session, not a one-off.

[RegreDanger]

Workaround that's been holding up for me (not a confirmed fix — I don't have visibility into the runner's Go source, so I can't say why this works, only that it has):

I suspected the root cause was specifically PowerShell's stdin read hanging in Antigravity's actual spawn context — $Input | Out-String in statusline.ps1/title.ps1 never returning in some invocations, which is presumably why the runner's kill path gets invoked at all (and then fails with TerminateProcess: Access is denied).

I tried adding a hard self-exit timeout inside the PowerShell script itself first (a background thread forcing [Environment]::Exit(0) after 30s regardless of where the script was stuck). It worked, but at a bad cost — it roughly 10x'd the per-invocation latency (~450ms → 3.4-7.5s), most likely from AMSI/Defender scrutinizing scripts that use low-level System.Threading APIs more heavily.

What I settled on instead: rewrote statusline.ps1/title.ps1 as small compiled Go binaries (statusline.exe/title.exe, same rendering logic, same JSON schema in) and pointed settings.json at those instead of powershell.exe -File .... No self-exit timer in the current version — I removed it in later testing and haven't seen the leak reproduce since.

To be clear about how confident I am in this: I could not cleanly re-trigger the original leak on demand afterward to prove the Go rewrite is what fixed it (my re-tests got confounded by manual testing in the same window). So take this as "switching off PowerShell for this specific hot path made the problem go away for me," not "I've proven why." If it's actually just a PowerShell-stdin-specific issue, this workaround should hold for anyone; if there's a deeper timing/luck element I haven't isolated, it might not generalize. Posting it here in case it helps someone in the meantime while the actual kill-path bug gets looked at.

[RegreDanger]

Update: confirmed the actual fix, and it wasn't what I initially thought.

I originally assumed rewriting statusline.ps1/title.ps1 as compiled Go binaries was enough on its own — I'd removed a self-exit safety timer during testing and didn't see the leak for a while, so I credited the PowerShell→Go switch itself.

That was wrong. Running the same Go binaries with no watchdog, the exact failure recurred, unprompted, during ordinary use (not a stress test):

⚠ Statusline error:
command failed: exec: canceling Cmd: TerminateProcess: Access is denied. (stderr: )

Same error as my original report. So the kill-path bug on the runner side isn't specific to PowerShell — it also affects a compiled Go binary. Whatever times out and fails to kill on the runner's end doesn't care about the child's language/runtime.

What actually holds up: a self-exit goroutine at the very top of main(), before anything else runs:

go func() {
    time.Sleep(5 * time.Second)
    os.Exit(0)
}()

This guarantees the process terminates on its own within 5s regardless of whether the runner's kill call succeeds. I run this in all three of my custom binaries (statusLine + title for agy, plus a second statusLine for an unrelated CLI) and haven't had an orphaned process since re-adding it.

Practical takeaway if you're hitting this: switching to a compiled binary alone won't fix it — you also need the binary to self-terminate on a timer, since the defect is in the runner's kill path, not in the child's runtime or language. I don't have visibility into the runner's Go source, so I still can't say why the kill fails, only that forcing the child to always exit on its own side-steps it completely.

[RegreDanger]

Another update — found the actual mechanism, and my last comment here was incomplete.

Short version of what changed: the self-exit watchdog I described before is not enough on its own, and now I know why.

The leak recurred after I posted my last update — same exact error, same binaries, watchdog compiled in and confirmed working in isolated testing. So I went and checked the thread state of a live zombie directly (Get-Process -Id <pid> | Select -Expand Threads | Select Id, ThreadState, WaitReason). All of its threads were Suspended — not blocked waiting on I/O, actually suspended by an external caller. A suspended thread can't run any code, including a timer goroutine sitting inside the same process.

So the actual sequence looks like this: the runner decides a child hasn't responded in time, suspends the whole process first, then calls TerminateProcess — which is the call that fails with Access is denied — and the process is left frozen forever, because nothing inside it can run anymore to save itself. A self-exit watchdog only works if it fires before the runner gets to that suspend step. Mine didn't, at least once, at a 5-second interval.

One more relevant data point: a plain Stop-Process -Force from an ordinary PowerShell session killed every one of these zombies without any trouble. So this isn't some deeper unkillable-process situation — it's specifically the handle/call the runner itself uses to terminate that's failing.

What I switched to instead of racing the clock: don't give the runner a reason to think the child is unresponsive in the first place. I replaced the blocking stdin read with a bounded one (~1s timeout) in the Go binary:

func readStdinWithTimeout(d time.Duration) []byte {
    ch := make(chan []byte, 1)
    go func() {
        data, _ := io.ReadAll(os.Stdin)
        ch <- data
    }()
    select {
    case data := <-ch:
        return data
    case <-time.After(d):
        return nil
    }
}

Now the process always returns within ~1s regardless of whether stdin ever closes, instead of blocking indefinitely and hoping a timer wins a race later. Kept the watchdog goroutine too, as a second layer, just in case something else blocks somewhere I haven't hit yet.

Holding clean so far, but I'm not going to claim this one's fully proven either — the original leak took about 50 hours to reach 565 processes, so a few hours of clean running isn't strong evidence yet. Updating here again if it recurs a third time; if it doesn't, I'll leave it as is.

[RegreDanger]

Cross-referencing #698 / #699 — looks like the same root cause I was missing the other half of.

akunzai's report there shows the runner itself holds the child's stdin pipe open without writing/closing it (during auth/resume warm-up, amplified by sub-agent fan-out like /code-review on a long-running session). On macOS that ends in a clean SIGKILL that just gets counted as a consecutive failure until the statusline auto-disables. On Windows, the same stdin-hold seems to end in the TerminateProcess: Access is denied I've been chasing — the kill itself fails instead of succeeding, which is why I got a frozen zombie instead of a clean disable.

Same underlying trigger, different failure mode per platform. And #699's fix for the example script is the same shape as what I landed on independently (bounded stdin read instead of a blocking one) — two people hitting this from opposite platforms and converging on the same workaround is a decent signal that the fix belongs upstream in the runner's stdin handling, not just in each of our scripts.