chore: raise DB slow-statement WARN threshold (1s -> 5s, configurable)#1162
Closed
frol-ai wants to merge 1 commit into
Closed
chore: raise DB slow-statement WARN threshold (1s -> 5s, configurable)#1162frol-ai wants to merge 1 commit into
frol-ai wants to merge 1 commit into
Conversation
sqlx warns on any statement over 1s, which is too sensitive here: the apalis-board admin pages run heavy full-table analytics over apalis.jobs — the `/api/v1/overview` and `/api/v1/queues` aggregates take ~1.4-1.5s — so every board load spams `WARN slow statement`. These are inherent to the board library (can't change their SQL) and run only on an admin page. Switch to `connect_with` and set the slow-statement threshold via `PgConnectOptions::log_slow_statements` (default 5s, `DB_SLOW_STATEMENT_THRESHOLD_SECONDS`). Routine multi-second analytics no longer warn, while genuinely pathological statements (the seconds-long token-list and backfill queries) still surface. The real speedup for the board is a smaller apalis.jobs table (retention + the stuck-Running reclaimer), tracked separately. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FRWEKUFYCcmGGhwdDYd471
Collaborator
Author
|
Superseded by #1163, which makes the board queries genuinely fast (de-bloating apalis.jobs) instead of raising the slow-statement threshold. Keeping sqlx's 1s threshold is actually preferable — it still flags the genuinely-slow statements (the ~6s silver upsert, ~12s token-list query). |
frol
pushed a commit
that referenced
this pull request
Jul 21, 2026
…1162) (#1163) Follow-up to your point on #1162 — instead of hiding the slow-statement warning by raising the threshold, actually make the board queries fast. ## Root cause: table bloat, not row count Measured on testenv: `TOTAL_JOBS` ≈ **70k** live rows, but `DB_SIZE` ≈ **371 MB** — ~5 KB/row, i.e. the table is almost all **dead tuples**. The queues churn ~1M rows/day (insert → Running → Done → pruned), and Postgres's default autovacuum only kicks in at 20% dead (`autovacuum_vacuum_scale_factor = 0.2`), which can't keep up. So the board's full-table aggregates (`SUM(CASE WHEN status …)`, `COUNT(*)`) scan hundreds of MB of mostly-dead pages → ~1.5s. ## Fix (attack the bloat) - **Aggressive autovacuum on `apalis.jobs`** (`tune_apalis_autovacuum`, run in `setup_apalis`): `vacuum_scale_factor = 0.05`, `autovacuum_vacuum_insert_scale_factor = 0.05` (the table is insert-heavy, so insert-triggered vacuums keep the visibility map fresh), lower thresholds, small cost delay. Keeps dead tuples low (compact heap) and the VM fresh (so the aggregates can use **index-only scans** over the narrow indexes already added, bypassing the heap entirely). - **`VACUUM (ANALYZE) apalis.jobs`** at the end of the hourly `apalis_prune`: reclaims the space the deletes leave + refreshes VM/stats. Plain `VACUUM` (not `FULL`) takes only a SHARE UPDATE EXCLUSIVE lock, so workers keep polling. ## Relationship to the other PRs - **Supersedes #1162** (slow-statement threshold bump) for these queries — recommend closing #1162, or keeping only as a minor defensive default. This PR makes them genuinely fast. - Complements #1161 (retention + stuck-`Running` reclaimer): fewer rows *and* no bloat. - **Ops, one-time**: `VACUUM FULL apalis.jobs` reclaims the existing ~370 MB immediately (brief exclusive lock); the settings here prevent it re-bloating. ## Testing `cargo fmt --check`, `cargo clippy --lib -- -D warnings`, `cargo check`: clean. Validated the `ALTER TABLE … SET (autovacuum_*)` and `VACUUM (ANALYZE)` statements against Postgres. 🤖 Generated with [Claude Code](https://claude.com/claude-code) https://claude.ai/code/session_01FRWEKUFYCcmGGhwdDYd471 Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Addresses the noisy
WARN slow statementlogs from the apalis-board endpoints:GET /api/v1/overview— ~1.4sGET /api/v1/queues— ~1.5sThese are the board library's full-table aggregate queries over
apalis.jobs(SUM(CASE WHEN status …)etc.), run only on the admin dashboard. Their SQL lives in the apalis-board crate (not ours to change), and 1.4–1.5s is expected for full-table analytics — but sqlx's default 1s slow-statement threshold flags them on every load, polluting the logs.Change
Switch the pool to
connect_withand set the threshold viaPgConnectOptions::log_slow_statements(Warn, …), default 5s (DB_SLOW_STATEMENT_THRESHOLD_SECONDS). Routine multi-second analytics no longer warn; genuinely pathological statements — the seconds-long token-listDISTINCT…UNIONand the backfill upserts — still warn (they exceed 5s).The real speedup for the board is a smaller
apalis.jobstable (retention + the stuck-Runningreclaimer in the companion PR); this change just fixes the log noise.Testing
cargo fmt --check,cargo clippy --lib -- -D warnings,cargo check: clean.Note: builds on the merged pool-hardening change (#1155); same pool builder.
🤖 Generated with Claude Code
https://claude.ai/code/session_01FRWEKUFYCcmGGhwdDYd471