fix: batch account-maintenance per cycle + reclaim stuck-Running tasks#1161
Merged
frol merged 1 commit intoJul 21, 2026
Merged
Conversation
…ng tasks Two related fixes for the phantom "10 account-maintenance tasks Running at once" (they were not concurrent — one worker, concurrency(1) — but orphaned Running rows, one per ~30 min = the handler `job_timeout`). 1. Batch the cycle. `run_maintenance_cycle` processed *every* enabled account, so total time was O(accounts); once it exceeded 30 min the handler timed out mid-cycle and the timed-out task was left `Running` (ack never landed). Now it processes at most `ACCOUNT_MAINTENANCE_BATCH_SIZE` (default 25) accounts, dirty first (never starved) then least-recently-synced (rotate across cycles). Worst-case cycle time is batch/concurrency × per-account timeout ≈ 12.5 min with defaults — comfortably under `job_timeout`. 2. Reclaim orphaned locks. apalis's orphan-reclaim only fires on a stale *worker* heartbeat, so a task stuck `Running` under a live worker (failed ack) is never reclaimed or pruned. `apalis_prune` now flips tasks stuck `Running` past a threshold (>= 2× `job_timeout`, so a real running task is never touched; `APALIS_STUCK_RUNNING_RECLAIM_SECONDS`) to `Killed`, which the retention sweep then removes. Validated the batch ordering and the reclaim UPDATE against a live schema (stuck row -> Killed, fresh row untouched). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FRWEKUFYCcmGGhwdDYd471
frol
pushed a commit
that referenced
this pull request
Jul 21, 2026
…1162) (#1163) Follow-up to your point on #1162 — instead of hiding the slow-statement warning by raising the threshold, actually make the board queries fast. ## Root cause: table bloat, not row count Measured on testenv: `TOTAL_JOBS` ≈ **70k** live rows, but `DB_SIZE` ≈ **371 MB** — ~5 KB/row, i.e. the table is almost all **dead tuples**. The queues churn ~1M rows/day (insert → Running → Done → pruned), and Postgres's default autovacuum only kicks in at 20% dead (`autovacuum_vacuum_scale_factor = 0.2`), which can't keep up. So the board's full-table aggregates (`SUM(CASE WHEN status …)`, `COUNT(*)`) scan hundreds of MB of mostly-dead pages → ~1.5s. ## Fix (attack the bloat) - **Aggressive autovacuum on `apalis.jobs`** (`tune_apalis_autovacuum`, run in `setup_apalis`): `vacuum_scale_factor = 0.05`, `autovacuum_vacuum_insert_scale_factor = 0.05` (the table is insert-heavy, so insert-triggered vacuums keep the visibility map fresh), lower thresholds, small cost delay. Keeps dead tuples low (compact heap) and the VM fresh (so the aggregates can use **index-only scans** over the narrow indexes already added, bypassing the heap entirely). - **`VACUUM (ANALYZE) apalis.jobs`** at the end of the hourly `apalis_prune`: reclaims the space the deletes leave + refreshes VM/stats. Plain `VACUUM` (not `FULL`) takes only a SHARE UPDATE EXCLUSIVE lock, so workers keep polling. ## Relationship to the other PRs - **Supersedes #1162** (slow-statement threshold bump) for these queries — recommend closing #1162, or keeping only as a minor defensive default. This PR makes them genuinely fast. - Complements #1161 (retention + stuck-`Running` reclaimer): fewer rows *and* no bloat. - **Ops, one-time**: `VACUUM FULL apalis.jobs` reclaims the existing ~370 MB immediately (brief exclusive lock); the settings here prevent it re-bloating. ## Testing `cargo fmt --check`, `cargo clippy --lib -- -D warnings`, `cargo check`: clean. Validated the `ALTER TABLE … SET (autovacuum_*)` and `VACUUM (ANALYZE)` statements against Postgres. 🤖 Generated with [Claude Code](https://claude.com/claude-code) https://claude.ai/code/session_01FRWEKUFYCcmGGhwdDYd471 Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
frol
approved these changes
Jul 21, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes the "multiple account-maintenance tasks Running at once" observed on the board. They were not concurrent — one worker,
concurrency(1), heartbeating fine — but orphanedRunningrows, one appearing every ~30 min (= the handlerjob_timeout).Root cause
run_maintenance_cycleprocessed every enabled account each tick, so the total time was O(accounts). Once that exceeded the 30-minjob_timeout, the handler was cancelled mid-cycle and the timed-out task was left inRunning— its ack to mark it terminal never landed (DB instability / cancelled handler). apalis's own orphan-reclaim only fires when the worker's heartbeat goes stale, but the worker was alive, so these phantomRunningrows were never reclaimed or pruned and just piled up.Fixes
ACCOUNT_MAINTENANCE_BATCH_SIZE, default 25). Dirty accounts sort first (never starved); the rest rotate bylast_synced_atacross cycles (cron fires every 60s). Worst-case cycle time =batch / concurrency × per_account_timeout≈ 25/4×120s ≈ 12.5 min — comfortably underjob_timeout, so cycles finish and ack cleanly.apalis_prune: flip tasks stuckRunningpast a threshold toKilled(then removed by the retention sweep). Threshold is at least 2×job_timeout(APALIS_STUCK_RUNNING_RECLAIM_SECONDS), so a genuinely-running task — bounded byjob_timeout— is never touched.Testing
cargo fmt --check,cargo clippy --lib -- -D warnings,cargo check: clean. Validated the batch ordering and the reclaim UPDATE against a live apalis schema (a 2h-stuckRunningrow →Killed; a 2-min-old one left untouched).Note: with #1156's leader election now merged, only the elected leader runs these — the batching keeps each leader's maintenance cycle bounded, and the reclaimer cleans up any locks orphaned during a leadership transition or DB blip.
🤖 Generated with Claude Code
https://claude.ai/code/session_01FRWEKUFYCcmGGhwdDYd471