Skip to content

Run scheduled maintenance securely on every site#1862

Open
stefan-burke wants to merge 82 commits into
mainfrom
work/scheduled-maintenance
Open

Run scheduled maintenance securely on every site#1862
stefan-burke wants to merge 82 commits into
mainfrom
work/scheduled-maintenance

Conversation

@stefan-burke

@stefan-burke stefan-burke commented Jul 18, 2026

Copy link
Copy Markdown
Member

What changed

  • Protect POST /scheduled with a unique key for each site before setup, routing, or database work.
  • Let an external monitor call each site directly instead of making the builder contact every child.
  • Generate and encrypt each child key, show it to owners, and let them send it to the child again after a failed provider write.
  • Run cleanup and activity-log backfill through one durable task list with leases, retry times, checkpoints, and request budgets.
  • Keep pending work inside the same 50-call request budget.
  • Retry pruning after one minute when a bounded delete fills its batch.
  • Page through encrypted invite expiries with a durable checkpoint and recheck invite state during deletion.
  • Retain new child sites before publishing them, including sites created by the local build command.
  • Support Bunny and Deno Deploy publication through their documented endpoints.

Key rotation

Coordinated one-click key rotation is intentionally deferred to the upcoming Uptime Kuma integration. That work will update the child site and its monitor together. Until then, a host operator can manually replace a compromised key on the child and in the monitor.

This PR does not include a second key slot, staged keys, promotion actions, or provider-specific rotation APIs.

Why

Quiet sites still need cleanup, but the old public endpoint could trigger database work and builder fan-out. Each site now has a small, bounded maintenance endpoint that a monitor can call safely.

Checks

  • Latest main is merged through 9e9fdef3; final review fixes are pushed through d28fb6e9.
  • Full precommit passed through the pinned Nix environment on the final tree.
  • Focused request-scope, scheduler, built-site, maintenance, pruning, local-build, users, provider, and query-scaling tests passed.
  • Targeted mutation reached 100% for every affected source, including 59/59 detected runner mutants and 28/28 scheduler mutants in the final review pass.
  • Branch-wide mutation checked 3,687 mutants: 3,656 killed, 31 proven equivalents, no survivors, and no timeouts.
  • Every review thread has an individual reply and is resolved.
  • Current GitHub checks are tracked on the latest pushed commit.

@coderabbitai

coderabbitai Bot commented Jul 18, 2026

Copy link
Copy Markdown
Contributor

Review Change Stack

Note

Reviews paused

It looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the reviews.auto_review.auto_pause_after_reviewed_commits setting.

Use the following commands to manage reviews:

  • @coderabbitai resume to resume automatic reviews.
  • @coderabbitai review to trigger a single review.

Use the checkboxes below for quick actions:

  • ▶️ Resume reviews
  • 🔍 Trigger review
📝 Walkthrough

Walkthrough

This PR replaces request-triggered pruning with durable scheduled maintenance, adds authenticated /scheduled execution, introduces per-site scheduler keys and provisioning, changes built-site publishing to retain records before publication, adds optimistic built-site updates, and instruments database and external calls with subrequest budgets.

Changes

Scheduled maintenance platform

Layer / File(s) Summary
Authenticated scheduled execution
src/shared/scheduled-access.ts, src/serve-app.ts, src/features/scheduled.ts, src/features/app/request.ts
Authorized POST /scheduled requests run registered maintenance tasks and return empty 204 or 503 responses with no-store caching. Safe successful reads can also trigger organic maintenance.
Durable maintenance and pruning
src/shared/db/migrations/*, src/shared/maintenance/*, src/shared/db/prune.ts, src/shared/db/activity-log-backfill.ts
Maintenance declarations, leases, migrations, bounded pruning, and activity-log backfill replace interval-marker scheduling.
Built-site scheduler lifecycle
src/shared/db/built-sites.ts, src/shared/site-scheduler.ts, src/shared/builder.ts, src/features/admin/built-sites.ts, src/ui/templates/admin/built-sites/panels.tsx
Built-site records store scheduler keys and revisions; providers prepare before publishing; admins can provision and verify child scheduler keys.
Request budgets and integrations
src/shared/subrequest-budget.ts, src/shared/db/query-log.ts, src/shared/fetch.ts, src/shared/storage.ts, src/shared/sentry-sdk.ts, src/shared/update.ts
Database and external operations are counted against nested request-scoped subrequest allowances.

Estimated code review effort: 5 (Critical) | ~120 minutes

Possibly related PRs

🚥 Pre-merge checks | ✅ 5
✅ Passed checks (5 passed)
Check name Status Explanation
Docstring Coverage ✅ Passed No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The title accurately summarizes the core change: secure, per-site scheduled maintenance.
✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch work/scheduled-maintenance

Comment @coderabbitai help to get the list of available commands.

@stefan-burke stefan-burke left a comment

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Submitting the pending review so the verified fixes can be replied to and resolved individually.

Comment thread README.md
Comment thread src/features/request-scopes.ts Outdated
Comment thread src/features/app/request.ts
Comment thread src/shared/builder.ts
Comment thread src/shared/bunny-cdn.ts Outdated
Comment thread src/shared/db/built-sites.ts Outdated
Comment thread src/shared/site-scheduler.ts
Comment thread src/shared/maintenance/registry.ts
Comment thread test/shared/db/built-site-scheduler.test.ts
Comment thread src/shared/db/prune.ts Outdated
@stefan-burke

Copy link
Copy Markdown
Member Author

Outside-diff finding: runner budget reserve. Resolved in 475535d. The literal is now TASK_RUNNER_CALL_RESERVE with a comment naming the claim, finish, and final no-work claim calls it protects. Runner tests preserve the existing capacity behavior.

@stefan-burke

Copy link
Copy Markdown
Member Author

Outside-diff finding: revalidate invite state at deletion. Resolved in 475535d. Every child and user DELETE now selects only ids that are still unactivated in the same batch, preventing an accepted invite from being removed after candidate discovery. The invite pruning tests cover the state-change race.

@stefan-burke

Copy link
Copy Markdown
Member Author

Outside-diff finding: storage upload content type. Resolved in 475535d. uploadRaw explicitly passes application/octet-stream to the Bunny SDK, and the request-level storage test verifies the upload behavior. The equivalent SDK normalization mutant is documented in 7d90730.

@stefan-burke

Copy link
Copy Markdown
Member Author

Outside-diff finding: bounded encrypted-invite scan. Resolved in 475535d. Invite pruning now reads at most MAINTENANCE_PRUNE_BATCH + 1 ordered candidates per run, persists the last id as its checkpoint, and decrypts only the bounded page. The large-current-prefix regression test proves forward progress without an unbounded scan.

@stefan-burke

Copy link
Copy Markdown
Member Author

Outside-diff finding: built-sites module size. Resolved in 475535d. Blob schema/validation, field mapping, and types moved into src/shared/db/built-sites/blob.ts, fields.ts, and types.ts; the I/O module is now 318 lines. Direct tests cover each extracted module.

@stefan-burke

Copy link
Copy Markdown
Member Author

Outside-diff finding: built-site panel hook. Verified and no change is needed. The beforeAll is inside describe("built site maintenance panel"), so it is suite-scoped rather than a module-level global BDD hook; it does not make the file ungroupable under the repository rule. Direct component coverage was strengthened in 5dabd8d.

@stefan-burke

Copy link
Copy Markdown
Member Author

Outside-diff finding: scheduled-key validation before persistence. Resolved in 475535d. buildSiteDataBlobFromInput parses the v2 blob with the isScheduledTaskKey check before serialization reaches the table insert/update. The blob regression tests reject invalid keys and cover valid/null values.

@stefan-burke

Copy link
Copy Markdown
Member Author

Outside-diff finding: equivalent-mutants merge conflict. Resolved by the branch merges through 0613331. The file contains the entries from both histories with no conflict markers; the final precommit, including lint, typecheck, duplicate detection, build, and full coverage suite, passed before 7d90730 was committed.

@stefan-burke stefan-burke left a comment

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Submitting resolved review-thread replies.

Comment thread src/shared/db/built-sites.ts Outdated
Comment thread src/shared/db/built-sites.ts
Comment thread src/shared/scheduled-access.ts Outdated
Comment thread src/serve-app.ts
Comment thread src/shared/site-scheduler.ts Outdated
Comment thread src/shared/deno-deploy-api.ts Outdated
Comment thread src/shared/maintenance/claims.ts Outdated
Comment thread scripts/build-site.ts

@stefan-burke stefan-burke left a comment

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Submitting resolved review-thread replies.

Comment thread test/shared/maintenance/runner.test.ts

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 9e9fdef3e4

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread src/shared/site-scheduler.ts
Comment thread src/shared/maintenance/runner.ts

@stefan-burke stefan-burke left a comment

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Submitting the final resolved review-thread replies.

Comment thread src/shared/maintenance/runner.ts

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: d28fb6e928

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

MAINTENANCE_REQUEST_CALL_LIMIT - used,
),
);
if (combinedAllowance === 0) return Promise.resolve();

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Reserve maintenance startup calls

When an organic GET has one subrequest left under the 40-call maintenance envelope, this zero-only guard still enters the runner. On a normal completed site, the registry checks activity_log_backfill.enabled with a DB read and then syncMaintenanceTaskRows performs a batched write before the loop can decide no task fits, so the second bookkeeping call exceeds the one-call allowance and reports a false organic-maintenance failure for an otherwise successful foreground request. Skip unless the remaining allowance can cover the runner's startup bookkeeping, not only when it is exactly zero.

Useful? React with 👍 / 👎.

if (!deployResult.ok) return deployResult;
return okResult({
defaultHostname: deployResult.value,
defaultHostname: `https://${createResult.value.slug}.deno.dev`,

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Use the Deno v2 production domain

For Deno-hosted builds this now stores https://<app>.deno.dev, but that is the old Deploy Classic shape; the current Deno Deploy docs describe v2 production domains as <app>.<organization>.deno.net (for example, my-app.acme-inc.deno.net). Any Deno child retained through this path gets an unreachable siteUrl, so assignment emails, renewal links, and scheduled-key verification target the wrong host even after the app publishes successfully.

Useful? React with 👍 / 👎.

return errorResult("Deploy code failed: no hostname in response");
}
return okResult(`https://${hostname}`);
return okResult(undefined);

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Wait for Deno revisions to succeed

With the request now going to the documented /deploy endpoint, Deno's v2 OpenAPI says this call returns 202 with a revision whose status can still be queued, building, or failed; this line treats that acceptance as a finished publish. For Deno-hosted builds and updates, a later build failure leaves the built-site row retained or the update flash successful even though no working revision was routed. Poll the revision/progress to a terminal succeeded state before returning success.

Useful? React with 👍 / 👎.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant