perf: de-bloat apalis.jobs so the board queries are fast (supersedes #1162)#1163
Merged
Merged
Conversation
…just quiet) The board's slow queries (`/api/v1/overview`, `/api/v1/queues` ~1.5s) aren't slow because of row count — testenv had only ~70k live rows — but because the table is **bloated**: ~370 MB of dead tuples behind that live set, so the full-table aggregates scan hundreds of MB. The queues churn ~1M rows/day and Postgres's default autovacuum (`scale_factor` 0.2) is far too lax to keep up. Fix the bloat at the source instead of hiding the warning: - `tune_apalis_autovacuum` (in `setup_apalis`): make autovacuum aggressive on `apalis.jobs` — `vacuum_scale_factor` 0.05, `insert_scale_factor` 0.05 (this table is insert-heavy), lower thresholds, small cost delay — so dead tuples are reclaimed promptly (heap stays compact) and the visibility map stays fresh (the aggregates can use index-only scans over the narrow indexes added earlier, bypassing the heap). - `VACUUM (ANALYZE) apalis.jobs` at the end of `apalis_prune` (hourly): reclaim the space the deletes leave and refresh the VM + stats. Plain VACUUM takes only a SHARE UPDATE EXCLUSIVE lock, so workers keep polling. This supersedes the slow-statement threshold bump (NEAR-DevHub#1162) for these queries — it makes them genuinely fast rather than silencing the warning. One-time `VACUUM FULL apalis.jobs` (ops, brief lock) reclaims the existing ~370 MB now; going forward the settings above keep it from re-bloating. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FRWEKUFYCcmGGhwdDYd471
frol
approved these changes
Jul 21, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Follow-up to your point on #1162 — instead of hiding the slow-statement warning by raising the threshold, actually make the board queries fast.
Root cause: table bloat, not row count
Measured on testenv:
TOTAL_JOBS≈ 70k live rows, butDB_SIZE≈ 371 MB — ~5 KB/row, i.e. the table is almost all dead tuples. The queues churn ~1M rows/day (insert → Running → Done → pruned), and Postgres's default autovacuum only kicks in at 20% dead (autovacuum_vacuum_scale_factor = 0.2), which can't keep up. So the board's full-table aggregates (SUM(CASE WHEN status …),COUNT(*)) scan hundreds of MB of mostly-dead pages → ~1.5s.Fix (attack the bloat)
apalis.jobs(tune_apalis_autovacuum, run insetup_apalis):vacuum_scale_factor = 0.05,autovacuum_vacuum_insert_scale_factor = 0.05(the table is insert-heavy, so insert-triggered vacuums keep the visibility map fresh), lower thresholds, small cost delay. Keeps dead tuples low (compact heap) and the VM fresh (so the aggregates can use index-only scans over the narrow indexes already added, bypassing the heap entirely).VACUUM (ANALYZE) apalis.jobsat the end of the hourlyapalis_prune: reclaims the space the deletes leave + refreshes VM/stats. PlainVACUUM(notFULL) takes only a SHARE UPDATE EXCLUSIVE lock, so workers keep polling.Relationship to the other PRs
Runningreclaimer): fewer rows and no bloat.VACUUM FULL apalis.jobsreclaims the existing ~370 MB immediately (brief exclusive lock); the settings here prevent it re-bloating.Testing
cargo fmt --check,cargo clippy --lib -- -D warnings,cargo check: clean. Validated theALTER TABLE … SET (autovacuum_*)andVACUUM (ANALYZE)statements against Postgres.🤖 Generated with Claude Code
https://claude.ai/code/session_01FRWEKUFYCcmGGhwdDYd471