feat(server): sticky provider-reported request size skips subsequent low-TPM models

[jasnoorgill]

Problem

When a provider rejects a chat request with a size error (Groq 413 'Limit N, Requested M', OpenRouter 400 context-length, Cloudflare 400/413), the same 36K-token prompt currently hits 4–5 models in the fallback chain sequentially. Each one rejects independently — wasting latency, burning per-key request budget on doomed attempts, and inflating the upstream 429 rate.

Live data (last 30 days, requests table where status='error'):

• groq: 2,385 size-related errors; 1,759 carry a parseable Requested N
• openrouter: 51 context-length errors; all parseable
• cloudflare: 25 (400 + 413); 18 parseable
• github: 564 (intentionally not parsed — see below)

The local token estimator (text.length / 4) undercounts tool-heavy / code-heavy prompts by 3–10×. A request the estimator scores at 3,700 tokens is reported as 36,532 by Groq — the models.tpm_limit pre-check passes, the call fires, and Groq 413s.

Solution

Once one upstream names the real token count for THIS request, propagate it as a sticky per-request value. Subsequent attempts in the same chain then pre-skip models whose TPM ceiling can't fit it — distinct from the existing canUseTokens headroom check (current usage + estimate vs limit) and surfaced as a new skip reason request-too-large-for-tpm so the exhaustion diagnostic tells the operator exactly why the model was skipped.

Implementation

File	Change
lib/provider-size-parser.ts (new)	Pure parser: parseProviderReportedSize(platform, message) → number | null. One regex per supported provider, all derived from the live error corpus.
lib/client-context.ts	Add observedRequestTokens to the AsyncLocalStorage context + setObservedRequestTokens / getObservedRequestTokens helpers. Sticky: never decreases (max wins).
lib/fallback-loop.ts	In recordRetryableFailure: call the parser; if it returns a number, write it via setObservedRequestTokens. Single site covers all three chat surfaces (proxy, responses, anthropic).
services/router.ts	In selectKeyForModel: new gate before canUseTokens — if observed > model's tpm_limit, skip with reason request-too-large-for-tpm.
__tests__/lib/provider-size-parser.test.ts (new)	16 cases: each supported provider with a real error body, plus edge cases (empty message, malformed number, GitHub Models limit-only).
__tests__/lib/client-context.test.ts	Update existing tests for the new field; add 4 cases for setObservedRequestTokens / getObservedRequestTokens (no-op outside scope, sticky max, per-request isolation).
__tests__/services/router.test.ts	Add 4 integration cases: skip when observed > tpm, surface new skip reason in exhaustion diagnostic, fall through when tpm is NULL, allow when observed fits.

Why GitHub Models is intentionally NOT parsed

GitHub's body is Request body too large for gpt-4.1 model. Max size: 8000 tokens. The 8000 is the LIMIT, not the rejected request's size. Returning it would cause every subsequent model with TPM < 8000 to be skipped for the rest of the request — wildly wrong since most free models have TPM ~6000. The parser returns null and learnLimitFromError keeps doing what it already does (lowering models.tpm_limit to 8000 for github/gpt-4.1).

Why AsyncLocalStorage and not a FallbackState field

The sticky value is request-scoped, not loop-scoped — the same store already carries IP/UA and is reset by clientContextMiddleware per request. Avoids threading a new arg through 7 function signatures (routeRequest, selectKeyForModel, runFallbackLoop, etc.). The router reads it on every iteration; the loop writes it once when a 4xx lands.

Backwards compatibility

• Opaque error bodies (bare "Request Entity Too Large", upstream 5xx without size payload, every provider not in the parser table) → null → no behavior change.
• canUseTokens headroom check still runs after the new gate.
• No DB schema change, no env vars, no new tables.
• All 949 existing tests still pass.

Verification

npx vitest run  # 949 passed, 0 failed
npm run build:server  # clean

Expected impact

• Eliminates ~8,000–14,000 wasted calls/month across Groq + OpenRouter + Cloudflare.
• One wasted call per request shape (the first 413) instead of one per model in the chain.
• New diagnostic reason request-too-large-for-tpm surfaces in /v1/docs exhaustion traces, so operators can see when a request was too big for the entire pool vs when it was just one model's 429.

Out of scope (follow-up ideas)

• Per-content-type estimator (prose vs code vs tool schemas). Would close most of the 3–10× undercount for tool-heavy prompts, so the gate is meaningful even when no provider has reported yet. Tracked separately because the parser-based gate already helps 90%+ of the wasted-call cases from the live data.
• OpenAI / Anthropic 413 parsers. No live 413 corpus for them in the requests table yet; parser slots are easy to add when the shape is observed.

[tashfeenahmed]

Solid work, and I like that the sizing decision is driven by what the provider actually reports rather than a guess. Two things before I go through it properly: it needs a rebase on current main (it's conflicting now), and Platform is redeclared locally — please import it from shared/types.ts instead so there's one definition.

Ping me once that's pushed and I'll review right after.

[jasnoorgill]

Rebased onto current main and addressed the Platform redeclaration.

What's in this push:

• Rebased onto origin/main (resolved conflicts in fallback-loop.ts, client-context.ts, services/router.ts, client-context.test.ts — all additive, kept HEAD's superset imports + your new observedRequestTokens field and methods).
• Platform now imported from @freellmapi/shared/types.ts — the local 12-union declaration is gone. Single canonical source.
• Build clean: npm run build -w server exits 0.
• Tests pass: 41/41 in the three affected files (provider-size-parser.test.ts, client-context.test.ts, router.test.ts).
• Full vitest suite: 1866/1868 passing; the 2 failures are pre-existing flaky timing tests in compression.test.ts (verified they fail on clean origin/main too, unrelated to this change).

Head SHA on this branch is now b3948f2. Ping me when you've taken a look and I'll address any further feedback.