A killed run reports 'running' forever and orphans state.json.lock, blocking all later dispatches

[andylxt]

Summary

When a bridge process dies without running its in-process handlers (hard kill, crash, host reboot), two pieces of cleanup are skipped. The second one bricks dispatch until someone deletes a file by hand.

Both are in scripts/lib/, plugin version 0.2.1.

1. Terminal status is never written, so a dead run reads running forever

Terminal status is written from the in-process path. A killed process never writes it, so the job record keeps status: "running" indefinitely and updatedAt stays frozen at whatever it was moments after start.

Observed: a run exited at 06:05:51 (its own log's last line is Grok finished.). At 06:18 the listing still showed running, phase starting, elapsed 21m. Its updatedAt was 1.7s after startedAt. Both pids in the record were gone.

The listing has no way to say "this is dead" — a running job and a killed job render identically. Scanning one workspace's state found three such zombie entries, the oldest a week old.

scripts/lib/process.mjs already exports processIsAlive. Only the stop path calls it. Calling it when rendering runs would resolve this without new machinery.

2. The orphaned lock is empty, so it can never be told from a held one

scripts/lib/state.mjs :: withStateLock:

fd = fs.openSync(lockPath, "wx");
...
} finally {
  fs.closeSync(fd);
  fs.unlinkSync(lockPath);
}

The finally is skipped on a hard kill, and nothing is ever written into the lock file — no pid, no timestamp. The acquire path has no staleness check either, so an orphaned lock is byte-identical to a held one.

Every later dispatch then fails after LOCK_MAX_ATTEMPTS (100) × LOCK_RETRY_MS (20ms) = ~2s:

Timed out acquiring state lock at .../state.json.lock

That error names the lock and never mentions the dead job that left it, so the natural read is "another dispatch is running" rather than "a previous one died". It cost three failed dispatch attempts and about 25 minutes before we traced it to a 0-byte file whose owning pid had been gone for four minutes.

Reproduction

1. Start any run.
2. taskkill /PID <bridgePid> /T /F (or kill -9) while it holds the lock.
3. runs still reports it running, indefinitely.
4. Every subsequent dispatch fails with Timed out acquiring state lock.
5. Deleting state.json.lock by hand restores dispatch.

Suggested fixes

• Write the owner pid and a timestamp into the lock file, and on acquire, treat a lock whose pid is gone (or which is older than some floor) as stale and reclaimable. An empty lock cannot be reasoned about by anything.
• Call the existing processIsAlive when rendering run status, so a non-terminal job with a dead pid reports as dead rather than as running.
• Optionally reconcile at startup: any non-terminal job whose pid is gone becomes failed.

The general shape: cleanup that only exists inside the process cannot survive that process being killed, which is exactly the case where it matters most.

Happy to send a PR if the approach looks right.