Skip to content

fix: time out stalled streaming error bodies - #286

Merged
franciscojavierarceo merged 1 commit into
vllm-project:mainfrom
StevenWang-CY:fix-streaming-error-body-timeout
Sep 11, 2026
Merged

fix: time out stalled streaming error bodies#286
franciscojavierarceo merged 1 commit into
vllm-project:mainfrom
StevenWang-CY:fix-streaming-error-body-timeout

Conversation

@StevenWang-CY

@StevenWang-CY StevenWang-CY commented Sep 11, 2026

Copy link
Copy Markdown
Contributor

Summary

A streaming request can hang after the upstream sends a 429/5xx response: successful SSE reads honor STREAMING_CHUNK_TIMEOUT_S, but reading an HTTP error body waits indefinitely. Reproduce with a one-second configured timeout and an upstream that sends 429 headers plus a partial diagnostic, then keeps the body open.

Apply the existing per-chunk timeout to error-body reads in executor-managed Responses streaming and Messages tool-loop requests, including initial and subsequent inference requests. The existing upstream-error object retains its status and processed headers, and adapters keep their current HTTP/SSE/WebSocket error formats. A timed-out diagnostic is discarded using the existing unreadable-body policy; a complete body is preserved. The timeout begins after response headers arrive, resets on each chunk, and remains disabled at zero. Non-streaming callers retain their existing timeout behavior.

Test Plan

  • The final regressions rebuilt against unchanged main produce six intended failures and five passing controls; all eleven pass with this patch.
  • Fresh production-binary comparison uses the real STREAMING_CHUNK_TIMEOUT_S environment variable: unchanged main remains pending beyond a 3.2-second watchdog, while the patch finishes in about one/two seconds for one/two-second settings and closes the stalled upstream body. Twelve cases per binary (24 total) also confirm zero timeout, progressing chunks, delayed headers, non-streaming compatibility, preserved errors, no extra inference, and empty failed-request storage.
  • Both regression binaries pass 50 fresh-process runs, including three concurrent workers: 175 reader and 100 transport/recovery test executions.
  • Reader coverage includes no body, partial body, progressing chunks, zero timeout, empty completion, cancellation, exact/oversized byte limits, invalid UTF-8, malformed JSON, and preserved status/headers.
  • Real HTTP Responses, WebSocket, and initial/later Messages requests terminate correctly. The later Messages case verifies a successful local search result in the second upstream request. Responses tests verify upstream body release, no failed-request persistence or implicit inference retry, then successful retry and actual continuation input after gateway/SQLite restart.
  • Exact clean commit: formatting, locked workspace/all-target/all-feature check and Clippy with warnings denied, 1,268 Rust tests, binary builds, launcher contracts, source-install CLI E2E, 115 Python tests, and 133 cassettes / 296 turns pass. Repository-wide pre-commit hooks pass.
  • All eight normally ignored PostgreSQL schema/storage tests also pass against a temporary PostgreSQL 17 instance; the instance is stopped after testing.
  • Hosted Rust CI confirms all eleven new regression tests pass.

Validation uses deterministic local upstreams and real HTTP/WebSocket gateways. Three installation-only Python tests and one existing Rust doctest remain skipped/ignored. No live model, GPU, cross-platform, or performance validation is claimed.

Signed-off-by: Chuyue Wang <stevenwang0805@outlook.com>
@franciscojavierarceo
franciscojavierarceo merged commit e87eeae into vllm-project:main Sep 11, 2026
14 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants