fix: back off cron-worker restarts so a DB blip can't cascade into a fleet stall

[frol-ai]

The catastrophic, non-recovering behavior in the logs traces to a hot restart loop.

Mechanism

A cron worker's backend is CronStream::new(schedule).pipe_to(store). apalis's pipe pushes ticks with a once future running send_all — a single store-push error ends it permanently. So when a DB connection reset kills a tick push, the pipe stream yields an error → the worker exits with WorkerError::StreamError. The Monitor's restart loop then rebuilds the worker immediately, with no delay (verified in apalis-core monitor::mod). During a persistent DB outage this is a tight loop: rebuild → push a tick → DB still down → exit → rebuild … which spins CPU, hammers the DB with reconnect attempts, and — on the shared tokio runtime — starves every other worker. That's how a brief DB blip cascaded into the fleet-wide stall that never recovered on its own.

Fix

Prepend an async backoff to the cron feed on restart (attempt > 0): 2s, 4s, 8s … capped at 60s. The delay stream yields no ticks, so no DB push is attempted until it elapses — turning the hot loop into exponential backoff. The worker now quietly retries until the DB recovers and resumes on its own, without a process restart.

The bronze public-history queue workers already self-recover (the poll fetcher backs off internally, 1s→5min); only the cron pipe lacked this.

Where this sits (incident series)

• #1157 — cut the recurring heavy query that pressures the DB (fewer crashes).
• #1155 — pool hardening: smaller connection footprint, survive resets.
• this — cron workers ride through a DB blip instead of hot-looping.
• #1154 (merged) — liveness self-heal: last-resort process restart + Sentry alert if a stall persists anyway.

Testing

cargo fmt --check, cargo clippy --lib --bins -- -D warnings: clean. Unit test for the backoff curve (0 on first start; 2s→60s cap on retries).

🤖 Generated with Claude Code

https://claude.ai/code/session_01FRWEKUFYCcmGGhwdDYd471