Fix: Reducing 429 errors due to RPM limits reached through dynamic delay

[ShubhanC]

What

When using agentic tools alongside freellmapi, we often run into an issue of reaching Requests per Minute (RPM) limits very quickly, due to repeated requests being sent rapidly. This PR implements a per-model rate limiter delay feature that applies request delays when usage exceeds provider thresholds. The throttler runs AFTER route resolution, so it sees the actual resolved model (not the client-sent model name), enabling accurate per-model rate limit enforcement.

The feature works by:

1. Reading the model's RPM/TPM limits from the database
2. Checking current usage via the sliding-window rate limit tracker
3. Applying a proportional delay when usage exceeds the per-provider threshold (default 50%)
   -- For example: Gemma4 26B from Google AI Studio has a max RPM of 10 and TPM of, so the delay will start kicking in once you reach 5 RPM requests.
4. Logging all throttle decisions for observability

Testing:

npm test
# or npx vitest run --pool=forks

The 2 test failures in analytics-requests.test.ts are pre-existing and unrelated to this PR (confirmed by running the same test on main branch, which fails identically).

Fixes

New Throttler Service (services/throttler.ts)

Implements per-model request throttling with the following features:

• Delay Formula: delay = max(100ms, floor((ratio - threshold) * 20000)) where ratio = used/limit

  • Below threshold → 0ms (no delay)
  • At threshold (e.g., 50%) → minimum 100ms
  • Above threshold → proportional delay (e.g., 75% usage with 0.5 threshold → 5,000ms)

• Dual-axis enforcement: Takes the MAX of RPM and TPM delays to ensure both axes are respected

• Per-provider thresholds: Reads from server/config/provider-limits.json (defaults to 0.5 for known providers, 0.8 for unknown)

• Graceful degradation: Returns 0 delay if model has no DB record, if rate limit status is unavailable, or if DB throws

• Logging: Every request logs either "pass" (below threshold) or "delay" (above threshold) with full context:

  [Throttler] 07:31:07 delay burst-test test-provider test-model rpm=45/60(75%) tpm=0/100000(0%) thresh=50% delay=5000ms
  

Integration Points

The throttler is called from the dispatch function in each route handler, right before the upstream provider call:

• routes/proxy.ts — POST /v1/chat/completions, POST /v1/completions
• routes/responses.ts — POST /v1/responses
• routes/anthropic.ts — POST /v1/messages

This ordering ensures the throttler sees the resolved model, not the client-sent model name — critical for auto-routed requests where the client sends model: "auto".

Tests

Unit Tests (services/throttler.test.ts) — 17 tests

• calculateDelay: boundary conditions, proportional delays, null limits, max of RPM/TPM
• checkThrottle: DB handling, logging, threshold variations (0.5, 0.8, 1.0, 0.99)
• applyThrottle: timing behavior with fake timers
• Edge cases: non-existent models, DB unavailability

Integration Tests (routes/throttler-integration.test.ts) — 8 tests

• Delay application with fake timers
• Cross-platform consistency
• Sequential request accumulation
• Different threshold scenarios

Load Tests (services/throttler-load.test.ts) — 4 tests

• Demonstrates 60% failure rate reduction (from 60% to 0%)
• Shows proper delay calculation and accumulation
• Validates throughput smoothing vs. fail-fast behavior

Performance Impact

While tracking quantitative change has been difficult to measure, the throttler transforms what would be hard failures (429 Too Many Requests) into managed delays that allow all requests to eventually succeed. I have some days where my success to failure ratio is 20:1, while most days are closer to 2:1 or 3:1. Before, most days with this tool had a ratio of 1:1, even with removal of bad keys and endpoints.

Found While Testing (Pre-existing, NOT fixed here)

analytics-requests.test.ts — Date boundary issue

Two tests in analytics-requests.test.ts fail with hardcoded dates from July 6, 2026:

expected +0 to be 3  // body.total = 0, expected 3
expected +0 to be 5  // body.total = 0, expected 5

Root cause: The test inserts data with created_at = '2026-07-06' but the query calculates since = now - 7 days. On July 14, 2026, this puts the cutoff at July 7 — so July 6 data is excluded by the WHERE clause.

Verified: The same 2 tests fail identically on main branch (1022 passed on main, including these failing).

npm run test -- src/__tests__/routes/analytics-requests.test.ts
# Main branch: Tests  2 failed | 1022 passed (1024)
# Throttler branch: Tests  2 failed | 1051 passed (1053)

This is a pre-existing test bug, not introduced by this PR.

Full Suite Results

Test Files  1 failed | 103 passed (104)
Tests  2 failed | 1051 passed (1053)
Duration  2.35s

All 29 throttler-specific tests pass. The 2 failures are pre-existing and unrelated to this PR.

Partially generated with Claude Code