Run scheduled maintenance securely on every site by stefan-burke · Pull Request #1862 · chobbledotcom/tickets

stefan-burke · 2026-07-18T18:54:33Z

What changed

Protect POST /scheduled with a unique key for each site before setup, routing, or database work.
Let an external monitor call each site directly instead of making the builder contact every child.
Generate and encrypt each child key, show it to owners, and let them send it to the child again after a failed provider write.
Run cleanup and activity-log backfill through one durable task list with leases, retry times, checkpoints, and request budgets.
Keep pending work inside the same 50-call request budget.
Retry pruning after one minute when a bounded delete fills its batch.
Page through encrypted invite expiries with a durable checkpoint and recheck invite state during deletion.
Retain new child sites before publishing them, including sites created by the local build command.
Support Bunny and Deno Deploy publication through their documented endpoints.

Key rotation

Coordinated one-click key rotation is intentionally deferred to the upcoming Uptime Kuma integration. That work will update the child site and its monitor together. Until then, a host operator can manually replace a compromised key on the child and in the monitor.

This PR does not include a second key slot, staged keys, promotion actions, or provider-specific rotation APIs.

Why

Quiet sites still need cleanup, but the old public endpoint could trigger database work and builder fan-out. Each site now has a small, bounded maintenance endpoint that a monitor can call safely.

Checks

Latest main is merged through 9e9fdef3; final review fixes are pushed through d28fb6e9.
Full precommit passed through the pinned Nix environment on the final tree.
Focused request-scope, scheduler, built-site, maintenance, pruning, local-build, users, provider, and query-scaling tests passed.
Targeted mutation reached 100% for every affected source, including 59/59 detected runner mutants and 28/28 scheduler mutants in the final review pass.
Branch-wide mutation checked 3,687 mutants: 3,656 killed, 31 proven equivalents, no survivors, and no timeouts.
Every review thread has an individual reply and is resolved.
Current GitHub checks are tracked on the latest pushed commit.

coderabbitai · 2026-07-18T18:54:40Z

Note

Reviews paused

It looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the reviews.auto_review.auto_pause_after_reviewed_commits setting.

Use the following commands to manage reviews:

@coderabbitai resume to resume automatic reviews.
@coderabbitai review to trigger a single review.

Use the checkboxes below for quick actions:

▶️ Resume reviews
🔍 Trigger review

📝 Walkthrough

Walkthrough

This PR replaces request-triggered pruning with durable scheduled maintenance, adds authenticated /scheduled execution, introduces per-site scheduler keys and provisioning, changes built-site publishing to retain records before publication, adds optimistic built-site updates, and instruments database and external calls with subrequest budgets.

Changes

Scheduled maintenance platform

Layer / File(s)	Summary
Authenticated scheduled execution `src/shared/scheduled-access.ts`, `src/serve-app.ts`, `src/features/scheduled.ts`, `src/features/app/request.ts`	Authorized `POST /scheduled` requests run registered maintenance tasks and return empty `204` or `503` responses with no-store caching. Safe successful reads can also trigger organic maintenance.
Durable maintenance and pruning `src/shared/db/migrations/`, `src/shared/maintenance/`, `src/shared/db/prune.ts`, `src/shared/db/activity-log-backfill.ts`	Maintenance declarations, leases, migrations, bounded pruning, and activity-log backfill replace interval-marker scheduling.
Built-site scheduler lifecycle `src/shared/db/built-sites.ts`, `src/shared/site-scheduler.ts`, `src/shared/builder.ts`, `src/features/admin/built-sites.ts`, `src/ui/templates/admin/built-sites/panels.tsx`	Built-site records store scheduler keys and revisions; providers prepare before publishing; admins can provision and verify child scheduler keys.
Request budgets and integrations `src/shared/subrequest-budget.ts`, `src/shared/db/query-log.ts`, `src/shared/fetch.ts`, `src/shared/storage.ts`, `src/shared/sentry-sdk.ts`, `src/shared/update.ts`	Database and external operations are counted against nested request-scoped subrequest allowances.

Estimated code review effort: 5 (Critical) | ~120 minutes

Possibly related PRs

chobbledotcom/tickets#1821: Shares dependent-delete logic used by database pruning.
chobbledotcom/tickets#1845: Overlaps in migration infrastructure and query logging.
chobbledotcom/tickets#1847: Overlaps in request routing and maintenance pipeline modules.

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title accurately summarizes the core change: secure, per-site scheduled maintenance.

✨ Finishing Touches

🧪 Generate unit tests (beta)

Create PR with unit tests
Commit unit tests in branch work/scheduled-maintenance

_{Comment @coderabbitai help to get the list of available commands.}

stefan-burke

Submitting the pending review so the verified fixes can be replied to and resolved individually.

stefan-burke · 2026-07-20T08:58:53Z

Outside-diff finding: runner budget reserve. Resolved in 475535d. The literal is now TASK_RUNNER_CALL_RESERVE with a comment naming the claim, finish, and final no-work claim calls it protects. Runner tests preserve the existing capacity behavior.

stefan-burke · 2026-07-20T08:58:53Z

Outside-diff finding: revalidate invite state at deletion. Resolved in 475535d. Every child and user DELETE now selects only ids that are still unactivated in the same batch, preventing an accepted invite from being removed after candidate discovery. The invite pruning tests cover the state-change race.

stefan-burke · 2026-07-20T08:58:53Z

Outside-diff finding: storage upload content type. Resolved in 475535d. uploadRaw explicitly passes application/octet-stream to the Bunny SDK, and the request-level storage test verifies the upload behavior. The equivalent SDK normalization mutant is documented in 7d90730.

stefan-burke · 2026-07-20T08:58:53Z

Outside-diff finding: bounded encrypted-invite scan. Resolved in 475535d. Invite pruning now reads at most MAINTENANCE_PRUNE_BATCH + 1 ordered candidates per run, persists the last id as its checkpoint, and decrypts only the bounded page. The large-current-prefix regression test proves forward progress without an unbounded scan.

stefan-burke · 2026-07-20T08:58:53Z

Outside-diff finding: built-sites module size. Resolved in 475535d. Blob schema/validation, field mapping, and types moved into src/shared/db/built-sites/blob.ts, fields.ts, and types.ts; the I/O module is now 318 lines. Direct tests cover each extracted module.

stefan-burke · 2026-07-20T08:58:54Z

Outside-diff finding: built-site panel hook. Verified and no change is needed. The beforeAll is inside describe("built site maintenance panel"), so it is suite-scoped rather than a module-level global BDD hook; it does not make the file ungroupable under the repository rule. Direct component coverage was strengthened in 5dabd8d.

stefan-burke · 2026-07-20T08:58:54Z

Outside-diff finding: scheduled-key validation before persistence. Resolved in 475535d. buildSiteDataBlobFromInput parses the v2 blob with the isScheduledTaskKey check before serialization reaches the table insert/update. The blob regression tests reject invalid keys and cover valid/null values.

stefan-burke · 2026-07-20T08:58:54Z

Outside-diff finding: equivalent-mutants merge conflict. Resolved by the branch merges through 0613331. The file contains the entries from both histories with no conflict markers; the final precommit, including lint, typecheck, duplicate detection, build, and full coverage suite, passed before 7d90730 was committed.

stefan-burke

Submitting resolved review-thread replies.

stefan-burke

Submitting resolved review-thread replies.

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 9e9fdef3e4

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

stefan-burke

Submitting the final resolved review-thread replies.

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: d28fb6e928

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 079d3b32f9

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 14e303ff0b

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

stefan-burke added 28 commits July 18, 2026 17:29

Add secure scheduled maintenance

374c59c

Map mutation tests to source files

57c74e1

Test built site action wrapper

d5f1e16

Split scheduler page mutation tests

0301b84

Move built site tests beside routes

d903e70

Test scheduled read-only blocking

d57ffc8

Test scheduled route isolation

ea84201

Test request scope wrapper

0578093

Test scheduled settings isolation

5d4212d

Test scheduler route definitions

796cdcf

Test API key attempt limiter

b30e6aa

Test successful login clearing

69a9438

Narrow login clearing test

26c67dd

Test historical prune migration

ef271c0

Test maintenance migration order

ef8af50

Test maintenance task schema

ceda86b

Test built site schema revision

47abe87

Test combined database budget

c74f291

Test retired maintenance settings

d432907

Move user tests beside source

d065b75

Test fetch subrequest counting

801370b

Test Sentry subrequest counting

44f9b86

Test retired settings keys

dc00f51

Test assignable site build

9eb728f

Test storage subrequest counting

db9f3d1

Test release subrequest counting

9326162

Test maintenance key panel

754495f

Test retired debug pruning state

9086232

Harden mutation contracts

efcad24

stefan-burke commented Jul 20, 2026

View reviewed changes

coderabbitai Bot approved these changes Jul 20, 2026

View reviewed changes

Merge origin/main into scheduled maintenance

9e9fdef

stefan-burke commented Jul 20, 2026

View reviewed changes

Comment thread test/shared/maintenance/runner.test.ts

chatgpt-codex-connector Bot reviewed Jul 20, 2026

View reviewed changes

Comment thread src/shared/site-scheduler.ts

Comment thread src/shared/maintenance/runner.ts

stefan-burke added 2 commits July 20, 2026 13:04

Stabilize listing query scaling test

7b88027

Keep scheduled retries within fresh state and budget

d28fb6e

stefan-burke commented Jul 20, 2026

View reviewed changes

Comment thread src/shared/maintenance/runner.ts

chatgpt-codex-connector Bot reviewed Jul 20, 2026

View reviewed changes

Comment thread src/shared/maintenance/runner.ts

Comment thread src/shared/deno-deploy-api.ts

Comment thread src/shared/deno-deploy-api.ts

stefan-burke added 2 commits July 20, 2026 14:48

Merge structured value validation from main

14ad188

Merge origin/main into scheduled maintenance

079d3b3

chatgpt-codex-connector Bot reviewed Jul 20, 2026

View reviewed changes

Comment thread src/shared/db/built-sites/blob.ts

Comment thread src/shared/site-db.ts

Merge origin/main into scheduled maintenance

14e303f

chatgpt-codex-connector Bot reviewed Jul 21, 2026

View reviewed changes

Comment thread src/shared/fetch.ts

Comment thread src/shared/maintenance/registry.ts

stefan-burke added this pull request to the merge queue Jul 21, 2026

Merged via the queue into main with commit 05fb17e Jul 21, 2026
5 checks passed

stefan-burke deleted the work/scheduled-maintenance branch July 21, 2026 17:26

This was referenced Jul 21, 2026

Finish scheduled maintenance safety checks #1880

Merged

Add tests for the Bunny CDN domain API #1879

Closed

Count Stripe calls and finish activity backfill once #1881

Merged

Uh oh!

Uh oh!

Conversation

stefan-burke commented Jul 18, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What changed

Key rotation

Why

Checks

Uh oh!

coderabbitai Bot commented Jul 18, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Reviews paused

Walkthrough

Changes

Uh oh!

stefan-burke left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

stefan-burke commented Jul 20, 2026

Uh oh!

stefan-burke commented Jul 20, 2026

Uh oh!

stefan-burke commented Jul 20, 2026

Uh oh!

stefan-burke commented Jul 20, 2026

Uh oh!

stefan-burke commented Jul 20, 2026

Uh oh!

stefan-burke commented Jul 20, 2026

Uh oh!

stefan-burke commented Jul 20, 2026

Uh oh!

stefan-burke commented Jul 20, 2026

Uh oh!

stefan-burke left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

stefan-burke left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

chatgpt-codex-connector Bot left a comment

Choose a reason for hiding this comment

💡 Codex Review

Uh oh!

Uh oh!

Uh oh!

stefan-burke left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

chatgpt-codex-connector Bot left a comment

Choose a reason for hiding this comment

💡 Codex Review

Uh oh!

Uh oh!

Uh oh!

Uh oh!

chatgpt-codex-connector Bot left a comment

Choose a reason for hiding this comment

💡 Codex Review

stefan-burke commented Jul 18, 2026 •

edited

Loading

coderabbitai Bot commented Jul 18, 2026 •

edited

Loading