bats suite flakes across the process-lifecycle tests: main itself fails ~half its runs, a different test each time

[Masashi-Ono0611]

What I measured

main itself fails roughly half its tests workflow runs, and a different test fails each time. Every failure I captured is a test that waits on a backgrounded process — its pid, its pidfile, or its death.

Last 8 tests runs on main:

When (UTC)	Result	Head	Test that failed
2026-08-01 19:03	failure	545973d0	launcher: a re-registered role gets a fresh child after deregistration (#485)
2026-08-01 18:44	success	870938e5	—
2026-08-01 07:22	failure	1c7efbc0	codex-monitor: never kills a non-codex process recorded under a reused pid
2026-08-01 06:33	success	cfa1a032	—
2026-08-01 06:00	success	303125f6	—
2026-07-31 08:54	failure	9140515b	marker-gc: drops a marker whose pid is ESRCH-dead
2026-07-31 08:17	success	6248bb05	—
2026-07-31 07:41	failure	39197bbe	launcher: a re-registered role gets a fresh child after deregistration

A fifth failure, this one from a PR run (#436, head 6bf5616a): watch: relaunch with the SAME instance id replaces the previous watcher (#66 preserved), failing at _wait_pidfile "$pf" "$w2" on the macos-latest 4/4 shard.

So: four distinct tests across four suites (test_launcher.bats, test_codex_monitor.bats, test_marker_gc.bats, test_watch.bats), and no single test that reproduces reliably.

Why I think these are one thing, not four

Each failing assertion is waiting for the OS to finish something the test cannot observe directly:

• launcher waits for a re-registered role's replacement child to exist
• codex-monitor waits for a recorded pid to be reused by a different process
• marker-gc waits for a pid to become ESRCH-dead
• watch waits for the pidfile to flip to the successor after the successor SIGTERMs the incumbent

That last one is instructive because the test already carries a comment describing a previous round of exactly this problem:

the successor SIGTERMs the old holder and then writes its own pid, so the pidfile can flip to w2 a beat before w1's TERM trap has run — poll for w1's exit rather than checking the instant the pidfile changes (the old single check raced this and flaked)

And #124 was the same shape in test_install.bats (watch.sh self-cleans a prior watcher on re-invocation, racing on the pidfile), closed after being fixed in place.

The pattern is that each race gets fixed where it is found. #503 and #541 both moved suites from fixed sleeps to condition waits, which is the right direction — but the remaining suites still have instances, and the ones above are what is left.

Why it costs something

The immediate cost is that a red X on a PR carries no information. On #436 the failing test is in tests/test_watch.bats; the PR touches scripts/delivery.sh, scripts/receiver-live.sh, and tests/test_delivery.bats. I had to pull four separate main run logs before I could tell whether I had broken something. A contributor without that habit either assumes their change is at fault and goes looking, or assumes it is not and stops reading CI.

Re-running the failed leg is the usual escape hatch, but a fork PR author has no workflow: write on the upstream repo — gh run rerun --failed returns Must have admin rights to Repository. So from a contributor's side the only ways out are pushing an empty commit or asking a maintainer.

What might help

I have not written a fix, so treat these as suggestions rather than a proposal:

• The four failures share a shape narrow enough to have one helper: wait until predicate P holds about pid N, with a deadline and a useful failure message. _wait_pidfile is already the local version of this in test_watch.bats. Hoisting one such helper into the shared test lib and converting the remaining process-lifecycle assertions to it would cover all four sites.
• Whatever the deadline is, the failure message wants to say what it was still waiting for and what it saw instead. _wait_pidfile ... failed does not distinguish "the successor never started" from "the successor started but the incumbent had not exited yet".
• If some of these are genuinely un-waitable (pid reuse in codex-monitor looks hard to make deterministic), marking them as such is more honest than leaving them in the gating set.

Happy to take one of these if it is useful — I would rather not start until the shape you want is settled, since the last round of this (#503, #541) was your call on the pattern.

Environment

Observed on GitHub-hosted runners, both ubuntu-latest and macos-latest legs, shard 4/4 in the cases I captured. Related to #124 (same class, already fixed in test_install.bats).