perf(ai-gateway): run the usage write in the primary's region

[RSO]

Summary

The AI gateway runs on kilocode-global-app, whose functions execute in both Frankfurt and SFO, while the PostgreSQL primary is Frankfurt-only. The usage write takes row locks on organization_user_usage and organizations (and kilocode_users for personal usage) and holds them across the remaining statements of the transaction until COMMIT. From SFO that hold is dominated by transatlantic round trips rather than by database work.

pg_stat_statements on the production primary shows the shape:

statement	calls	mean	max
UPDATE organizations SET microdollars_used …	5.0M	0.5 ms	100,590 ms
INSERT organization_user_usage … ON CONFLICT DO UPDATE	6.0M	0.30 ms	100,242 ms
the microdollar_usage_ins CTE	338M	2–10 ms	~119,900 ms

A sub-millisecond statement taking 100 seconds is queueing, not work, and the CTE's max is clipped exactly at the database statement_timeout ceiling of 120 s. Measured arrival rate on a single hot counter row peaked at 9 writes/second against a hold of a few hundred milliseconds, so the queue grows without bound. Each waiter pins a pool connection while it waits, and with max: 10 per instance ten of them take out every route on that instance — which is what surfaces in Sentry as timeout exceeded when trying to connect on unrelated endpoints like /api/profile and /api/webhooks/github.

SFO instances now hand the write to POST /api/internal/usage/record on APP_URL, the Frankfurt-only kilocode-app deployment whose rewrites do not divert /api/internal/*. Frankfurt instances keep writing directly, since a Frankfurt-to-Frankfurt hop is pure overhead and a pointless new failure mode.

The seam is saveUsageRelatedData. The caller keeps stream parsing, cost computation and NUL sanitization, and ships the already-normalized { core, metadata } from toInsertableDbUsageRecord. That payload carries no prompt or response bodies.

Correctness details worth knowing:

• Idempotent on core.id, which the sender generates and which is the microdollar_usage primary key. Without it, a caller retry after a lost response would collide on the primary key and report a billed request as unbilled.
• Nullable contract fields use .nullable(), never .optional(). JSON.stringify drops undefined keys, and null is load-bearing here: org usage sets the prompt prefixes to null precisely so prompt text is never persisted.
• created_at is validated as strict ISO 8601 so a PostgreSQL-shaped timestamp can never enter the path, per packages/db/AGENTS.md. Covered by fixtures using production-shaped text.
• Failure degrades to the local write. On any non-retryable response or exhausted retries the client returns unavailable and the caller falls back. A slow billing record beats a lost one; the failure goes to Sentry with the usageId so a genuine loss is reconcilable.

No new environment variables — APP_URL and INTERNAL_API_SECRET already exist.

The first commit is an unrelated documentation addition recording the Frankfurt/SFO function-region split and which observability fields actually name the compute region. It is separable if you'd rather it went on its own.

Verification

Automated: tsgo --noEmit -p apps/web/tsconfig.json clean; ./scripts/lint-all.sh 0 warnings 0 errors; 29 new tests passing; regression runs of src/lib/ai-gateway + src/lib/drizzle.test.ts (626 tests, 54 suites), src/lib/kilo-pass-org + organization-usage (127 tests), and spend-writer-audit. This is targeted verification, not pnpm validate.

Manual: none. The behaviour that matters only exists on a multi-region production deployment — locally VERCEL_REGION is unset, so isUSRegion() is false and the code takes the pre-existing local path. I could not exercise the SFO branch, the HTTP hop, or the dedupe path by hand.

• Confirm INTERNAL_API_SECRET is present on kilocode-global-app. If it is missing, every SFO write silently takes the fallback path and nothing improves.
• Confirm app.kilo.ai resolves to kilocode-app. If that domain ever moves to the global project this silently becomes a no-op hop.
• Exercise POST /api/internal/usage/record against a preview deployment, including sending the same core.id twice to see status: "duplicate".
• After deploy, watch recordUsageInPrimaryRegion failures in Sentry and the max_exec_time of UPDATE organizations before/after.

Visual Changes

N/A

Reviewer Notes

Please push back on the premise. I have not proven that the waiters are each other. The mean-vs-max evidence is statistical; I never captured pg_locks or wait events during a burst. A single long-running transaction holding an organizations row — an admin script, a cron, a batch job — would produce identical maxima with no convoy, and in that case this change does not help. A pg_stat_activity + pg_blocking_pids() sample during an incident would settle it and is cheap to add to the existing db-pool-metrics cron. I'd rather that ran before this is relied on.

Known gaps in test coverage:

• No test for the region-gating wiring. saveUsageRelatedData is module-private and reaching it means going through countAndStoreUsage, which needs a Response. isUSRegion and the client are tested separately, but the seam between them is not.
• No integration test for the endpoint, including the dedupe branch. That is the riskiest new logic and deserves a route test against the test database.

Other things to look at:

• There is precedent in this repo against self-referencing HTTP fetches within apps/web (see the comments in lib/auto-fix/github/get-fix-config.ts). This is cross-deployment rather than same-instance, which is why the hop buys locality — but it is a fair objection and the isUSRegion() gate exists partly to answer it.
• maxDuration = 150 on the new route is deliberately above the 120 s database statement_timeout so a blocked write is reported rather than truncated into a lost billing row.
• Retries can double-deliver while the server is mid-write. The dedupe check handles the committed case; the uncommitted-but-in-flight case relies on the primary key, where the loser 500s and the caller's next attempt sees duplicate.
• This shortens the hot-row hold but does not remove the hot row. organization_user_usage is one row per (org, user, day) and organizations.microdollars_used is shared by every member. Batching in a single place is the natural follow-up, and the endpoint is now that place.