Skip to content

fix(core): fail over immediately on rate limits in FallbackModel chains - #117

Merged
berges99 merged 1 commit into
mainfrom
fix/fallback-rate-limit-fail-fast
Aug 7, 2026
Merged

fix(core): fail over immediately on rate limits in FallbackModel chains#117
berges99 merged 1 commit into
mainfrom
fix/fallback-rate-limit-fail-fast

Conversation

@berges99

@berges99 berges99 commented Aug 7, 2026

Copy link
Copy Markdown
Collaborator

A 429 with a large Retry-After made _retry_on_error sleep for the full provider cooldown (unbounded, up to minutes) before exhausting retries and handing off — turning "fallback in <1s" into multi-minute stalls.

  • Cap retry sleeps at MAX_RETRY_DELAY (30s) even when Retry-After is larger
  • New fail_fast_rate_limit: raise on 429/RateLimitError without retrying; FallbackModel sets it for every entry that still has a fallback behind it
  • Last entry in the chain (and single models) retry in place as before

A 429 with a large Retry-After made _retry_on_error sleep for the full
provider cooldown (unbounded, up to minutes) before exhausting retries and
handing off — turning "fallback in <1s" into multi-minute stalls.

- Cap retry sleeps at MAX_RETRY_DELAY (30s) even when Retry-After is larger
- New fail_fast_rate_limit: raise on 429/RateLimitError without retrying;
  FallbackModel sets it for every entry that still has a fallback behind it
- Last entry in the chain (and single models) retry in place as before
@cursor

cursor Bot commented Aug 7, 2026

Copy link
Copy Markdown

PR Summary

Medium Risk
Touches core LLM retry timing and fallback failover on rate limits; behavior changes for 429 handling in multi-model chains and for any caller using large Retry-After values.

Overview
Fixes multi-minute stalls when a provider returns 429 with a large Retry-After: in-place retries could sleep for the full header value before fallback could run.

_retry_on_error now caps any retry sleep (including Retry-After floors) at MAX_RETRY_DELAY (30s). It also accepts fail_fast_rate_limit: when set, rate limits (RateLimitError / 429) re-raise immediately with no sleep or in-place retries; other retryable errors (e.g. 503) behave as before.

FallbackModel passes fail_fast_rate_limit=True for every chain entry that still has a fallback behind it, and False on the last entry so the final model still retries in place. _llm_router threads the flag into the shared retry loop.

Tests cover huge Retry-After capping, fail-fast on 429, non-rate-limit retries with fail-fast enabled, and per-entry flags on a three-model chain.

Reviewed by Cursor Bugbot for commit 88142ea. Bugbot is set up for automated code reviews on this repo. Configure here.

@berges99
berges99 merged commit 782b7e7 into main Aug 7, 2026
11 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants