Run scheduled maintenance securely on every site

README.md

@@ -24,12 +24,13 @@ This is the recommended way to deploy it. Fork the repo, connect it to Bunny, an
    | `DB_URL`            | Your Bunny database URL            |
    | `DB_TOKEN`          | Your Bunny database auth token     |
    | `DB_ENCRYPTION_KEY` | 32-byte base64-encoded AES-256 key |
+   | `SCHEDULED_TASK_KEY` | Unique 32-byte base64url maintenance key |
 
 5. **Add GitHub Actions secrets** to your repository: `BUNNY_SCRIPT_ID` and `BUNNY_ACCESS_KEY`
 
 Pushes to `main` trigger the deploy workflow automatically. The database schema auto-migrates on first request. Visit `/setup/` to set your admin password and currency.
 
-For image uploads, also add `STORAGE_ZONE_NAME` and `STORAGE_ZONE_KEY` as Bunny secrets. See the [CONFIG_KEYS reference](https://chobbledotcom.github.io/tickets/doc.ts/~/CONFIG_KEYS.html) for all optional variables.
+For image uploads, also add `STORAGE_ZONE_NAME` and `STORAGE_ZONE_KEY` as Bunny secrets. Configure an external monitor using the [scheduled maintenance guide](docs/scheduled-maintenance.md). See the [CONFIG_KEYS reference](https://chobbledotcom.github.io/tickets/doc.ts/~/CONFIG_KEYS.html) for all optional variables.
 
 ---
 
@@ -355,7 +356,7 @@ Optional:
 | `ADMIN_EMAIL_ADDRESS` | Enables a superuser recovery account. The email local-part (before `@`) must be a valid username: 2–32 characters, letters, numbers, hyphens, and underscores only. Email delivery must be configured before the superuser can be enabled. |
 | `DEBUG_KEY` | Optional diagnostic key. `GET /health` returns a plain `Up :)` by default; a request carrying a matching `X-Debug-Key` header gets a small JSON payload (build commit, build timestamp, server time). Unset ⇒ the verbose response is disabled. |
 
-**Database maintenance:** pruning of expired sessions, rate-limit rows, payment idempotency records and (optionally) orphaned attendees runs automatically while serving requests, self-gated to roughly once per `PRUNE_INTERVAL_HOURS` (default 24) per table — so a site with regular traffic needs no setup. To guarantee pruning on a quiet site, point a cron at `GET /scheduled` — a dynamic route that prunes on every hit (static asset URLs such as `/favicon.ico` are served before pruning runs, so they won't do). On a builder, `POST /scheduled` additionally pokes the least-recently-pruned built site (a plain request that triggers _its_ prune), so one cron on the master keeps quiet client sites pruned too — run it often enough to cover the fleet within `PRUNE_INTERVAL_HOURS` (e.g. hourly handles ~24 clients at the default).
+**Database maintenance:** pruning of expired sessions, rate-limit rows, payment records, and optional orphan attendees runs automatically while serving requests. For quiet sites, configure an external monitor to send an authenticated `POST /scheduled` to each site at least every 15 minutes. Each site needs its own key. The builder does not contact child sites for the monitor. See the [scheduled maintenance guide](docs/scheduled-maintenance.md) for setup and CDN rules.
 
 **Backups:** every table is dumped to a single `.zip`, with table reads keyset-paginated so no single response trips libsqld's "Response is too large" payload cap (the server limit behind Bunny's databases). Backups run **out-of-band**, not inside the migration: a full dump of a ~31-table schema can't fit alongside a migration within one edge request's [50-subrequest budget](https://docs.bunny.net/scripting/limits), so migrations just migrate, and a backup is taken by GitHub Actions (or `deno task backup`) beforehand. To enforce that, **`/admin/update` and the per-site update button refuse to deploy unless a backup of that database was taken in the last hour.**
 

TODO.md

@@ -448,7 +448,7 @@ look.
 
 ## Request performance: consolidate AsyncLocalStorage scopes
 
-`src/features/index.ts` enters eleven nested request scopes for locale, client
+`src/features/app/request.ts` enters eleven nested request scopes for locale, client
 IP, request ID, request cache, query logging, flash, session memoization, iframe
 mode, CSRF, saved form data, and settings auditing. Replace them with one typed
 `RequestContext` in one `AsyncLocalStorage`; retain domain methods where they add

deno.json

@@ -52,7 +52,7 @@
     "#shared/": "./src/shared/",
     "#i18n": "./src/shared/i18n.ts",
     "#locales/": "./src/locales/",
-    "#routes": "./src/features/index.ts",
+    "#routes": "./src/features/app/request.ts",
     "#routes/": "./src/features/",
     "#templates/": "./src/ui/templates/",
     "#static/": "./src/ui/static/",

docs/scheduled-maintenance.md

@@ -0,0 +1,49 @@
+# Scheduled maintenance
+
+Each site runs its own small database maintenance jobs. An external HTTPS
+monitor must send one request to each site at least every 15 minutes:
+
+```http
+POST /scheduled HTTP/1.1
+Authorization: Bearer <site key>
+```
+
+The request has no body. A successful run returns an empty `204`. A configured
+site returns an empty `401` for a missing or wrong key. A site with no key, or a
+non-`POST` request, returns an empty `404`. A system failure returns an empty
+`503`. All responses use `Cache-Control: no-store`.
+
+## Set Up A Site
+
+Create a different 32-byte key for every independently deployed site:
+
+```bash
+openssl rand -base64 32 | tr '+/' '-_' | tr -d '='
+```
+
+Store it as the native `SCHEDULED_TASK_KEY` secret on that site. Never put the
+key in a URL, monitor name, note, log, or plaintext database field. The owner
+can read the local key on **Settings > Advanced**.
+
+The built-site manager creates a different key for every child. It stores the
+key in the child's native secret and in the builder's encrypted site data. Use
+the child's **Scheduled maintenance** tab to set up an older child.
+
+## Change A Child Key
+
+Coordinated one-click key rotation is intentionally deferred to the upcoming
+Uptime Kuma integration, which will update the child and its monitor together.
+Until then, a host operator can manually replace a compromised key on the child
+and in the monitor.
+
+## CDN Rules
+
+Allow the monitor to reach `/scheduled` without a browser challenge, cached
+response, redirect, or body rewrite. If the CDN has an allowlist, add the
+monitor there.
+
+Rate-limit `/scheduled` at the CDN before requests reach the edge script. The
+application deliberately does not keep a request counter for rejected calls:
+doing so would let unauthenticated traffic consume database subrequests before
+authentication. Keep the limit high enough for the monitor's retries, but far
+below a general API traffic rate.

scripts/build-site.ts

@@ -12,7 +12,7 @@
  *   BUNNY_API_KEY=... deno run --allow-all scripts/build-site.ts "My Event Site"
  */
 
-import { builderApi } from "#shared/builder.ts";
+import { buildRetainedSite } from "#shared/site-build.ts";
 import { runBuildEdge } from "./run-build-edge.ts";
 
 const [siteName] = Deno.args;
@@ -41,7 +41,7 @@ console.log(
   `Bundle ready (${code.length} bytes). Provisioning site "${siteName}"…`,
 );
 
-const result = await builderApi.buildSite({ code, siteName });
+const { result } = await buildRetainedSite(siteName, { code, siteName });
 
 if (!result.ok) {
   console.error(`Build failed: ${result.error}`);

scripts/find-unused-src.ts

@@ -205,7 +205,6 @@ const entryPoints = new Set([
   "src/index.ts", // deno task start
   "src/edge.ts", // esbuild entry for Bunny CDN
   "src/fp.ts", // import map root alias
-  "src/features/index.ts", // import map root alias
   "src/doc.ts", // deno doc generation
   "src/shared/jsx/jsx-dev-runtime.ts", // jsxImportSource compiler config
 ]);

scripts/mutation/equivalent-mutants.txt

@@ -45,10 +45,10 @@ src/shared/config.ts:177:33  ?? → ||    # getEnv(): string|undefined, only fal
 src/shared/qr-token.ts:60:18  ?? → ||    # input.date?: string, only falsy string "" === fallback ""
 src/shared/qr-token.ts:62:18  ?? → ||    # input.name?: string, only falsy string "" === fallback ""
 src/shared/app-forms.ts:71:18  ?? → ||    # config.auth is an AuthPolicy object when present, so it is always truthy
-src/shared/site-assignment.ts:192:57  ?? → ||    # parseReadOnlyFromMs(): number|null, 0 ?? 0 === 0 || 0
-src/shared/site-assignment.ts:370:34  ?? → ||    # available[idx]: object|undefined, never falsy non-null
-src/shared/site-assignment.ts:392:34  ?? → ||    # getEmailConfig(): EmailConfig|null
-src/shared/site-assignment.ts:416:53  ?? → ||    # parseEmail(): ValidEmail|null, always truthy or null
+src/shared/site-assignment.ts:161:57  ?? → ||    # parseReadOnlyFromMs(): number|null, 0 ?? 0 === 0 || 0
+src/shared/site-assignment.ts:332:34  ?? → ||    # available[idx]: object|undefined, never falsy non-null
+src/shared/site-assignment.ts:354:34  ?? → ||    # getEmailConfig(): EmailConfig|null
+src/shared/site-assignment.ts:378:53  ?? → ||    # parseEmail(): ValidEmail|null, always truthy or null
 src/shared/ledger/project.ts:20:41  ?? → ||    # allBalances: Map<string,number>.get; the only falsy-non-null number is 0, and 0 ?? 0 === 0 || 0
 src/shared/listing-parents-rules.ts:124:4  ?? → ||    # find()?.error(): an i18n message is never "", so only undefined reaches the fallback either way
 src/shared/ledger/project.ts:38:49  ?? → ||    # balanceOf: same Map<string,number>.get fallback; 0 ?? 0 === 0 || 0
@@ -69,7 +69,21 @@ src/shared/accounting/rows.ts:262:58  ?? → ||   # selectById (await …)[0]: T
 src/shared/accounting/rows.ts:133:72  ?? → ||   # renderInsert guard?.args: InValue[]|undefined — an array is always truthy
 src/shared/accounting/manual-entries.ts:209:29  ?? → ||   # guard error message `transfer.kind ?? ""`: string|undefined, "" ?? "" === "" || ""
 src/shared/checkout-ledger.ts:35:72  ?? → ||   # find(...)?.amount: number|undefined; the only falsy-non-null amount is 0 and 0 ?? 0 === 0 || 0
-src/features/settings-bundles.ts:264:40  ?? → ||   # bundle is a readonly string[] or undefined; every present array, including [], is truthy
+src/features/settings-bundles.ts:247:40  ?? → ||   # bundle is a readonly string[] or undefined; every present array, including [], is truthy
+src/shared/scheduled-access.ts:16:20  ?? → ||   # the optional regex capture is non-empty because the capture uses +, so only undefined reaches the null fallback
+src/shared/maintenance/runner.ts:113:40  ?? → ||   # wakePolicy is a non-empty MaintenanceWakePolicy value or undefined, so only undefined reaches the fallback
+src/shared/builder.ts:201:44  ?? → ||   # dbToken is string|undefined and the fallback is "", so its only falsy string already equals the fallback
+src/shared/builder.ts:202:35  ?? → ||   # dbProvider is a non-empty provider value or undefined, so only undefined reaches the fallback
+src/shared/builder.ts:207:36  ?? → ||   # dbProvider is a non-empty provider value or undefined, so only undefined reaches the fallback
+src/shared/builder.ts:287:26  ?? → ||   # hostingProvider is a non-empty provider value or undefined, so only undefined reaches the fallback
+src/features/url.ts:58:35  ?? → ||   # URLSearchParams.get returns null or a string; its only falsy string is already the empty fallback
+src/shared/storage.ts:376:18  application/octet-stream → ""   # the pinned Bunny SDK normalizes an empty contentType to application/octet-stream, as the request-level upload test proves
+src/shared/site-build.ts:39:21  = → +=   # retainedId starts at 0 and buildSite invokes retain exactly once, so both assignments store row.id
+src/shared/db/built-site-scheduler.ts:11:36  ?? → ||   # scheduledTaskKey is null or a non-empty canonical key, so both operators select the candidate only when no key exists
+src/shared/deno-deploy-api.ts:85:34  ?? → ||   # env_vars is a record object when present, and every object is truthy
+src/shared/bunny-cdn.ts:477:42  ?? → ||   # DefaultHostname is string|undefined; its only falsy string is already the same empty fallback
+src/shared/bunny-cdn.ts:529:50  ?? → ||   # Secrets is an array|null, and every present array is truthy while null takes the fallback either way
+src/shared/subrequest-budget.ts:38:40  ?? → ||   # scoped counts is an object when present, and every object is truthy
 src/shared/db/attendees/balance.ts:223:22  ?? → ||   # attendee status ids are positive integers when present, so the value is always truthy or nullish
 src/shared/accounting/queries.ts:162:26  || → &&   # transferActivityBounds: MIN(occurred_at) and MAX(occurred_at) over one table are NULL together (both iff the table is empty), so either-null and both-null coincide
 src/shared/admin-features.ts:95:58  ?? → ||   # find(): AdminFeatureDefinition|undefined; a feature object is always truthy
@@ -114,6 +128,12 @@ src/shared/db/table.ts:715:6  === → ==   # wrapNullable constrains T to exclud
 # Login attempt rows store an integer count. For undefined, null, zero, or any
 # non-zero integer, `(attempts ?? 0) + 1` and `(attempts || 0) + 1` agree.
 src/shared/db/login-attempts.ts:46:39  ?? → ||
+src/shared/db/users.ts:263:62  ?? → ||   # queryAll()[0] is a truthy UserAuthFields object or undefined, so both operators return the row or null
+src/shared/db/query-log.ts:71:62  ?? → ||   # queryLogScope.current() is a truthy QueryLogState object or undefined
+src/shared/db/query-log.ts:252:43  ?? → ||   # stored read counts start at 1 and remain positive, so only undefined reaches the zero fallback
+src/shared/db/query-log.ts:292:22  ?? → ||   # store is a truthy QueryLogState object or undefined
+src/shared/update.ts:131:45  ?? → ||   # the stored marker is a string or undefined and the fallback is empty, so both operators return the same string
+src/shared/update.ts:156:45  ?? → ||   # the stored commit is a string or undefined and the fallback is empty, so both operators return the same string
 
 # Parent/child fold moved to fold-tree.ts in Phase 2a (see the fold-tree.ts
 # entries below); the ticket-payment.ts fold-internal equivalents went with it.
@@ -243,11 +263,6 @@ src/shared/db/attendees/capacity/groups.ts:178:39  ?? → ||   # membership.get(
 # v2 signed metadata (Phase 2d).
 src/shared/payment-helpers.ts:159:24  ?? → ||    # toBookingItems intent.allocations: ChildAllocation[]|undefined — an array is always truthy, undefined→[] either way
 
-# Settings registry: readOnly is optional but can only be true when present, so
-# `"readOnly" in accessor && accessor.readOnly` and `... || accessor.readOnly`
-# make the same if-condition decision for every valid accessor.
-src/shared/settings/registry.ts:345:40  && → ||   # readOnly?: true; absent is falsy either way, present is true either way
-
 # --- External order widget (src/ui/client/order.ts) ----------------------------
 # URL matches and catalog entries are either non-empty strings/objects or absent,
 # so these nullish and truthy fallbacks have the same result.
@@ -389,14 +404,11 @@ src/shared/images/resize.ts:75:19  = → +=    # out[o+1] = G into fresh-zeroed
 src/shared/images/resize.ts:76:19  = → +=    # out[o+2] = B into fresh-zeroed array; written once
 src/shared/images/resize.ts:78:17  = → +=    # out[o+3] = A into fresh-zeroed array; written once
 
-# middleware.ts's missing Content-Type fallback is fed only to startsWith checks,
-# so neither "" nor "mutated" can match an accepted MIME prefix. Once a tracking
-# key is found, assigning true with `=` or `+=` is the same (`false + true` is 1,
-# which is truthy), and continuing instead of breaking can only rediscover more
-# tracking keys without changing the final boolean.
-src/features/middleware.ts:163:63   → "mutated"    # neither fallback matches an accepted Content-Type
-src/features/middleware.ts:213:18  = → +=    # false + true is truthy, matching true
-src/features/middleware.ts:214:7  break; → (removed)    # later keys cannot reverse hasTracking
+# middleware.ts's Content-Type fallback already equals the only falsy string.
+# Once a tracking key is found, continuing instead of breaking can only
+# rediscover more tracking keys without changing the final boolean.
+src/features/middleware.ts:155:39  ?? → ||    # Content-Type is string|null; the only falsy non-null string is "", which is also the fallback
+src/features/middleware.ts:217:7  break; → (removed)    # later keys cannot reverse hasTracking
 
 # bundle-loader.ts assigns query/style/script values onto freshly-created URLs
 # and elements. Each left-hand value is empty (or false for defer), so `=` and
@@ -939,8 +951,8 @@ src/features/admin/index.ts:28:21  ?? → ||   # `path.split("/")[2] ?? ""`: the
 src/features/admin/index.ts:45:43  ?? → ||   # areasBySegment values are arrays, so every present value is truthy and a miss is undefined
 
 # App prefix dispatch (app/routes.ts) — two provably-equivalent survivors.
-src/features/app/routes.ts:239:14   → "mutated"   # publicPagePath only receives the declared prefixes "", "listings", and "terms"; for "", both ternary arms return "/", and neither non-empty prefix equals "mutated", so no declared route path changes
-src/features/app/routes.ts:444:59  ?? → ||   # route handlers return Response|null; every Response object is truthy, while null (and the mutation runner's return-undefined mutants) selects notFoundResponse under both operators
+src/features/app/routes.ts:235:14   → "mutated"   # publicPagePath only receives the declared prefixes "", "listings", and "terms"; for "", both ternary arms return "/", and neither non-empty prefix equals "mutated", so no declared route path changes
+src/features/app/routes.ts:439:59  ?? → ||   # route handlers return Response|null; every Response object is truthy, while null (and the mutation runner's return-undefined mutants) selects notFoundResponse under both operators
 
 # Logger (logger.ts) — two provably-equivalent survivors from the run over the
 # logger suites.

src/features/admin/builder.ts

@@ -22,10 +22,11 @@ import {
   isTursoEnabled,
 } from "#shared/config.ts";
 import { logActivity } from "#shared/db/activityLog.ts";
+import { providerOrBunny } from "#shared/db/built-sites/types.ts";
 import {
   builtSites,
+  builtSitesCrudTable,
   insertBuiltSite,
-  providerOrBunny,
 } from "#shared/db/built-sites.ts";
 import { settings } from "#shared/db/settings.ts";
 import { getEnv } from "#shared/env.ts";
@@ -158,32 +159,46 @@ const builderPost = createAuthedFormRoute({
     const dbError = dbProviderConfigError(dbProviderVal, values.db_url);
     if (dbError) return errorRedirect(BUILDER_PATH, dbError);
 
+    let retainedSiteId: number | null = null;
+    const assignable = form.getFlag("assignable");
     const result = await settings.withCurrentTask("builder", () =>
-      builderApi.buildSite({
-        ...(dbProviderVal === "manual" ? {} : { dbProvider }),
-        dbToken: values.db_token,
-        dbUrl: values.db_url,
-        hostingProvider,
-        siteName: values.site_name,
-      }),
+      builderApi.buildSite(
+        {
+          ...(dbProviderVal === "manual" ? {} : { dbProvider }),
+          dbToken: values.db_token,
+          dbUrl: values.db_url,
+          hostingProvider,
+          siteName: values.site_name,
+        },
+        async (prepared) => {
+          const row = await insertBuiltSite(
+            values.site_name,
+            prepared.defaultHostname,
+            prepared.dbUrl,
+            prepared.dbToken,
+            false,
+            prepared.hostingId,
+            undefined,
+            prepared.hostingProvider,
+            prepared.dbProvider,
+            prepared.scheduledTaskKey,
+          );
+          retainedSiteId = row.id;
+        },
+      ),
     );
 
     if (!result.ok) return errorRedirect(BUILDER_PATH, result.error);
 
     const buildResult = result.value;
     if (!buildResult.ok) return errorRedirect(BUILDER_PATH, buildResult.error);
 
-    await insertBuiltSite(
-      values.site_name,
-      buildResult.defaultHostname,
-      buildResult.dbUrl,
-      buildResult.dbToken,
-      form.getFlag("assignable"),
-      buildResult.hostingId,
-      undefined,
-      buildResult.hostingProvider,
-      buildResult.dbProvider,
-    );
+    if (retainedSiteId === null) {
+      throw new Error("Built site was published before it was retained");
+    }
+    if (assignable) {
+      await builtSitesCrudTable.update(retainedSiteId, { assignable: true });
+    }
     await logActivity(`Built new site: ${values.site_name}`);
 
     return redirect(