Poison pill detection and automatic quarantine

[LeoBorai]

Problem

A .wasm binary that causes the Wasmtime executor to panic or trap before it can write any
result will be re-queued by the Reaper indefinitely, creating an infinite crash loop that
consumes executor capacity and pollutes logs with noise.

Proposed Solution

Track a separate recovery_count column incremented by the Reaper each time it re-queues a
job (distinct from attempt, which is incremented at claim time). If recovery_count reaches
a configurable threshold (default 3) the job is moved to dead with an explanatory error
message, regardless of remaining max_attempts.

ALTER TABLE jobs
    ADD COLUMN recovery_count INT NOT NULL DEFAULT 0;

Reaper update for re-queued jobs:

UPDATE jobs
SET
    status         = 'pending',
    claimed_at     = NULL,
    claimed_by     = NULL,
    heartbeat_at   = NULL,
    recovery_count = recovery_count + 1
WHERE status     = 'running'
  AND COALESCE(heartbeat_at, claimed_at) < NOW() - visibility_timeout
  AND attempt    < max_attempts
  AND recovery_count < $1   -- MATE_POISON_PILL_THRESHOLD
RETURNING id;
 
-- Quarantine poison pills
UPDATE jobs
SET
    status = 'dead',
    result = '{"error":"poison pill: executor crashed without recording a result 3 times"}'::jsonb
WHERE status         = 'running'
  AND COALESCE(heartbeat_at, claimed_at) < NOW() - visibility_timeout
  AND recovery_count >= $1;

The threshold should be configurable via MATE_POISON_PILL_THRESHOLD (default 3).

Acceptance Criteria

• recovery_count column added via migration.
• Reaper increments recovery_count on each re-queue (not on normal retry).
• Job is moved to dead with a descriptive result JSON when recovery_count reaches
  the configured threshold.
• MATE_POISON_PILL_THRESHOLD environment variable is documented in .env.example.
• Integration test: a job that never writes a result is quarantined after N reaper cycles.