Problem
Once a job is moved to running, nothing currently re-queues it if the worker crashes,
is OOM-killed, or loses its database connection. The job is stranded in running state
indefinitely.
Proposed Solution
Add a visibility_timeout column (default 5 minutes) and a Reaper background task
that runs on a fixed interval. The Reaper re-queues any running job whose claimed_at
is older than its visibility_timeout without having completed, up to max_attempts. Jobs
that exceed max_attempts are moved to dead.
-- Re-queue stale jobs that still have retries remaining
UPDATE jobs
SET
status = 'pending',
claimed_at = NULL,
claimed_by = NULL
WHERE status = 'running'
AND claimed_at < NOW() - visibility_timeout
AND attempt < max_attempts;
-- Move to dead if retries exhausted
UPDATE jobs
SET status = 'dead'
WHERE status = 'running'
AND claimed_at < NOW() - visibility_timeout
AND attempt >= max_attempts;
The Reaper should run as a tokio::spawn-ed loop inside the server process and fire every
60 seconds (configurable via environment variable MATE_REAPER_INTERVAL_SECS).
Schema Changes
ALTER TABLE jobs
ADD COLUMN visibility_timeout INTERVAL NOT NULL DEFAULT '5 minutes';
Acceptance Criteria
Problem
Once a job is moved to
running, nothing currently re-queues it if the worker crashes,is OOM-killed, or loses its database connection. The job is stranded in
runningstateindefinitely.
Proposed Solution
Add a
visibility_timeoutcolumn (default5 minutes) and a Reaper background taskthat runs on a fixed interval. The Reaper re-queues any
runningjob whoseclaimed_atis older than its
visibility_timeoutwithout having completed, up tomax_attempts. Jobsthat exceed
max_attemptsare moved todead.The Reaper should run as a
tokio::spawn-ed loop inside the server process and fire every60seconds (configurable via environment variableMATE_REAPER_INTERVAL_SECS).Schema Changes
Acceptance Criteria
visibility_timeoutcolumn added via migration with a5 minutesdefault.MATE_REAPER_INTERVAL_SECS(default60).runningis moved back topendingaftervisibility_timeoutelapses.max_attemptswithout completing is moved todead, not re-queued.is re-queued within one reaper cycle.