Skip to content

Crash recovery via claimed_at timeout and a Reaper task #148

Description

@LeoBorai

Problem

Once a job is moved to running, nothing currently re-queues it if the worker crashes,
is OOM-killed, or loses its database connection. The job is stranded in running state
indefinitely.

Proposed Solution

Add a visibility_timeout column (default 5 minutes) and a Reaper background task
that runs on a fixed interval. The Reaper re-queues any running job whose claimed_at
is older than its visibility_timeout without having completed, up to max_attempts. Jobs
that exceed max_attempts are moved to dead.

-- Re-queue stale jobs that still have retries remaining
UPDATE jobs
SET
    status       = 'pending',
    claimed_at   = NULL,
    claimed_by   = NULL
WHERE status      = 'running'
  AND claimed_at  < NOW() - visibility_timeout
  AND attempt     < max_attempts;
 
-- Move to dead if retries exhausted
UPDATE jobs
SET status = 'dead'
WHERE status     = 'running'
  AND claimed_at < NOW() - visibility_timeout
  AND attempt    >= max_attempts;

The Reaper should run as a tokio::spawn-ed loop inside the server process and fire every
60 seconds (configurable via environment variable MATE_REAPER_INTERVAL_SECS).

Schema Changes

ALTER TABLE jobs
    ADD COLUMN visibility_timeout INTERVAL NOT NULL DEFAULT '5 minutes';

Acceptance Criteria

  • visibility_timeout column added via migration with a 5 minutes default.
  • Reaper task starts automatically when the server starts.
  • Reaper interval is configurable via MATE_REAPER_INTERVAL_SECS (default 60).
  • A job stranded in running is moved back to pending after visibility_timeout elapses.
  • A job that reaches max_attempts without completing is moved to dead, not re-queued.
  • Integration test: simulate a crash (drop the connection mid-execution) and assert the job
    is re-queued within one reaper cycle.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions