Issue batch: local-Ollama cooldown fix, keyless-config boot crash, served-model drift guard, failover-ladder UI, credits - #603
Merged
Merged
Conversation
A declarative config written when a platform was still keyless (pollinations lost `keyless: true` in #573) carries a `keys` entry with no `key`. applyDeclarativeConfigFromEnv() runs inside main(), and upsertApiKey threw `key is required for <platform>`, so main().catch exited 1 — a stale config bricked the install at startup. Boot-time apply now degrades that shape to a per-entry skip: any non-keyless platform entry without a key logs a warning naming the platform (with a remediation hint — pollinations points at enter.pollinations.ai) and is skipped, while the rest of the config still applies. The result gains a `warnings` array and the boot summary reports the skip count. Genuinely malformed config (unparseable JSON, schema violations, unknown platforms, custom entries without baseUrl) still fails loudly, and the write-time API path in routes/keys.ts is untouched. Fixes the boot-crash from #600 (legacy keyless pollinations config). Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Credit reporters and a proposer from adopted issues: - @NirvanaCh7 (#549 GitHub Models retirement heads-up; #335 routing/ observability write-up that led to same-model unification in #341) - @Mohamed3nan (#335 comment proposing same-model-first failover) - @Arman-Espiar (#592 local Ollama routes hitting false 429 cooldowns) - @MetaMysteries8 (#600 Pollinations community models + keyless boot crash) - @lujun880726 (#499 disabled models stuck in Fusion slots) @LoneRifle (#534 Kilo auto-route aliasing) is already on the wall for #179/#161, so no new avatar. Also document the previously undocumented routing strategy suffixes in "Using the API": auto:smart | auto:fast | auto:cheap | auto:reliable | auto:balanced, their synonyms and case-insensitivity, and auto:<profile-name> for routing through a named profile (server/src/services/router.ts GLOBAL_SORT_ALIASES + resolveRoutingChain). Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Local inference routes (custom-platform Ollama/llama.cpp/LM Studio) were getting fake 429s: a slow generation timed out, classified retryable, and fed the null-limits exhaustion heuristic — 90s transient bench, then the 2m -> 10m -> 1h -> 24h ladder — stranding what is usually the user's ONLY route and rendering "All models exhausted". Three changes at the existing choke points: - Error-kind gating: the null-limits "effectively daily-exhausted" heuristic in getCooldownDecisionForLimit now only fires on an actual quota signal (new isRateLimitSignal: structured 429 or rate-limit wording). Timeouts and 5xx keep the short transient bench and no longer record heuristic hits. cooldownDecisionForError and fusion's decision sites thread the error kind through; the Ollama-Cloud protection (real 429s on opaque-quota providers escalating) is preserved and now tested. - Locality exemption: a key whose base_url points at loopback/RFC1918 (new sync isLoopbackOrPrivateUrl in url-guard — literal IPs and 'localhost' only, no DNS on the hot path) is capped at a 5s bench with source 'heuristic', never the ladder, even on a literal 429 or a Retry-After. Verdict cached per key id (base_url is immutable per row). - Ladder decay: cooldownHits now clears on a successful request, like nullLimitHits — a served request proves the quota is alive, so the next failure starts the ladder over instead of inheriting up-to-24h steps from stale hits. Escalation while a quota is genuinely spent is untouched (no successes can occur then), and the documented ladder contract keeps its coverage. 17 new tests (red-green via patch method); full suite 1369 passing. Fixes #592. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The proxy overwrites the upstream `model` field with the routed id before
the caller sees it (the gateway contract — providers like Reka return
placeholders such as "default"). That overwrite also destroyed the only
evidence when a provider silently serves a DIFFERENT model than requested.
Nothing in the codebase detected served != requested.
- Capture the raw upstream model BEFORE the overwrite, on both the
non-streaming path (result.model) and the streaming path (first frame
that carries one). Covers openai-compat, cohere, and cloudflare, whose
adapters pass the upstream body through; google/aihorde synthesize their
responses with the routed id, so they carry no upstream signal.
- lib/served-model.ts compares served vs routed ignoring cosmetic spelling
(case, provider/org prefix, ":free"-style tier suffix, separator runs)
and placeholders ("default"), warn-logs `[ServedModel] platform=...
requested=... served=...` once per (platform, requested, served) tuple
per process, and returns the raw served id for persistence.
- New nullable requests.served_model column (migration
20260726_000005_request_served_model) written via lib/request-log.ts
only when the served model genuinely differs — NULL in the healthy case
keeps the table small and drifted rows trivially queryable.
- Per-request analytics detail payload now includes servedModel.
The response contract is unchanged: clients still see the routed model in
bodies and streamed frames. Tests: 17 new (unit normalization/log-cap +
integration for both paths, cosmetic equality, NULL column, contract);
full suite 1369 passing, zero regressions from the 1352 baseline.
Served-model drift detection for #534.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Server:
- GET /api/analytics/requests accepts optional status ('success'|'error')
and platform query filters; both validated (400 on bad input), applied as
bound parameters, and absent filters keep the behavior byte-identical.
- New migration 20260726_000005_attempt_error_summary: nullable
request_attempts.error_summary. Populated by the fallback loop for every
attempt that ended on an error — sanitized through lib/error-redaction.ts
(same secret/URL scrubbing as the parent row's error text) and capped at
200 chars via the new summarizeAttemptError(). NULL for ok/committed hops
and pre-existing rows. GET /api/analytics/requests/:id now returns
errorSummary per attempt.
Client (Analytics page):
- Recent calls: renders the previously-dropped attemptCount, adds
status (All/Success/Errors) + provider filters in the panel header, and
makes rows clickable.
- Clicking a row opens a request-detail dialog (the app's dialog idiom):
parent-row fields plus the failover ladder — ordinal, platform (with its
routing.ts color), model, key ordinal, outcome badge, start-offset +
duration, and the redacted per-hop error summary when present.
- New per-provider breakdown table surfacing the successRate and errorCount
the by-platform endpoint already returned (plus p95/TTFT/tok-s/tokens).
- All strings added to all six locales (en, es, fr, it, pt-BR, zh-CN).
Tests: +7 (filter validation/behavior, error_summary persistence with
redaction + cap, detail payload, summarizeAttemptError); full suite 1359
passing, client build clean.
Failover-ladder drill-down + provider stats UI — closes the remaining slice of #335.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…uest_served_model)
This was referenced Jul 26, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes for the top verified open issues, plus the observability UI that closes #335's remaining slice.
fix: local Ollama false rate-limits (#592)
The null-limits exhaustion heuristic (written for Ollama Cloud's opaque quota) benched local endpoints 90s→24h on mere timeouts. Now: the heuristic only escalates on genuine rate-limit signals (429/quota), never timeouts/5xx; keys with loopback/RFC1918 base_urls cap at a 5s bench; the escalation ladder decays on success. Ollama Cloud protection explicitly preserved (locality decided by base_url).
fix: keyless declarative config bricks boot (#600)
A legacy config entry for a formerly-keyless platform (pollinations, post-#573) crashed the server at startup. Missing-key entries now skip with an actionable warning; malformed config still fails loudly. Live-verified on a production-image container: old code exit-1'd, new code boots and applies the rest of the config.
feat: served-model drift detection (#534)
Kilo's auto-routes silently serve arbitrary models (live-verified twice in one day: Gemma in the morning, stepfun/step-3.7-flash by afternoon). The gateway's model-field overwrite (correct, kept) also destroyed the only evidence. Now the upstream claim is captured pre-overwrite, normalized (case/namespace/tier-suffix; placeholder values like Reka's "default" ignored), warn-logged once per tuple, and persisted in
requests.served_model+ the analytics detail payload.feat: failover-ladder drill-down + provider stats UI (closes the last slice of #335)
Analytics page: attempts column, click-through dialog rendering the full per-attempt ladder (platform, model, key ordinal, outcome, timing, redacted error summary — new
request_attempts.error_summary), status/provider filters (server-side), and a per-provider breakdown table surfacing the already-computed successRate/errorCount. 6 locales.docs
README: five contributor credits (reporters of #549, #592, #600, #499, #335 + the same-model-first proposal) and the previously undocumented
auto:smart|fast|cheap|reliable|balanced/auto:<profile>routing syntax.Testing