Worker can remain "running" with a stale runner and stop processing work #12

Activity

1. aford22 commented on Sep 7, 2026

   

   aford22

   on Sep 7, 2026

   More actions

   Independent reproduction — without the 0.4.7 → 0.4.9 upgrade

   We hit this on @convex-dev/batch-worker@0.2.1 (via @convex-dev/workpool@0.4.10) and root-caused it to the same mechanism before finding this issue. Posting our evidence in case a second affected user helps move #13 along.

   One data point that may narrow the cause: we never ran workpool 0.4.7 or 0.4.9. This project went from 0.2.18 directly to ^0.4.10, so the 0.4.7 → 0.4.9 upgrade named as the suspected trigger is not necessary to reproduce. That suggests the stale-running window is reachable in normal operation rather than only across that specific version transition.

   What we observed (measured 2026-09-06, local Convex deployment):

   • workflowPool's worker reported status: {"kind":"running"} while its loop was dead.
   • Both the loop runner and the monitor were absent from _scheduled_functions. Since the watchdog is only re-armed from inside a loop/monitor run, once both vanish in the same window nothing can restart either — which matches "the monitor was also stale, so it did not recover the worker".
   • Every enqueue → ping no-opped on the stale flag. Work accumulated to 469 items with inProgress: 0, with no error logged anywhere.
   • A process restart does not clear it — the wedge is persisted database state. We restarted twice before realising this.
   • Other pools in the same deployment were unaffected (idle, no runner/monitor ids), so it was isolated to the one worker.

   Impact. The failure is silent and surfaces far from its cause. Ours presented as end-to-end tests timing out in fixture setup, waiting on a workflow that never dispatched; two engineers independently suspected their own test code first. Downstream it left 68 half-created organisations that could not be cleaned up, because the workflow that completes their setup never ran.

   Recovery — same as yours, confirming forceKick works:

   npx convex run --component <pool>/batchWorker lib:stop  '{"name":"main"}'
   npx convex run --component <pool>/batchWorker lib:start '{"name":"main"}'
   

   Backlog drained 471 → 358 → 234 → 130 → 1 and the worker returned to idle.

   On #13: the approach there — checking runner liveness inside ping() rather than trusting the stored status — looks right to us, and is materially better than the kick() added in 0.3.0 for this failure mode. kick() still needs a human who has already noticed; a liveness check in ping() means the next enqueue recovers the worker automatically. We'd rather not build a watchdog around a bug that has an authored fix. Is there anything blocking #13 being un-drafted, and is a 0.2.x backport plausible? Workpool currently constrains batch-worker to ^0.2.2, so 0.3.x isn't reachable for workpool consumers without an override.

   Happy to test a pre-release against our reproduction.