[Backend] Worker queue jobs silently dropped when Redis connection is lost mid-flight

[ekelemepraise-code]

Summary

When the Redis connection is interrupted after a job has been dequeued but before it is acknowledged, the job is neither retried nor moved to the dead letter queue. It is silently dropped.

Observed Behaviour

During a Redis failover (e.g., Sentinel promotion), jobs in the PROCESSING state are lost. There is no log entry, no metric, and no alert.

Expected Behaviour

1. All in-flight jobs should be tracked with a heartbeat/lease mechanism
2. Jobs whose leases expire should be requeued automatically
3. Jobs that exceed max_retries should be routed to the DLQ
4. A worker.job.dropped metric should be emitted

Implementation Notes

• Relevant file: backend/src/workers/
• Consider using Apalis' built-in retry policies
• DLQ routing was partially scoped in [BACKEND] Implement Dead Letter Queues (DLQ) for Async Workers #689 but not connected to the lease expiry path

Acceptance Criteria

• Lease/heartbeat mechanism implemented for all worker job types
• Expired leases trigger requeue within configurable window
• DLQ routing wired for max-retry-exceeded jobs
• worker.job.dropped metric emitted and tested