Skip to content

fix: skip unhealthy workers in proxy GET endpoints - #232

Open
Bevisy wants to merge 1 commit into
vllm-project:mainfrom
Bevisy:fix/proxy-get-healthy-worker
Open

fix: skip unhealthy workers in proxy GET endpoints#232
Bevisy wants to merge 1 commit into
vllm-project:mainfrom
Bevisy:fix/proxy-get-healthy-worker

Conversation

@Bevisy

@Bevisy Bevisy commented Sep 2, 2026

Copy link
Copy Markdown

Purpose

Fixes #135/v1/models (and other proxy GET endpoints) returns 500 when any worker is unhealthy.

select_first_worker returned the first registered worker without checking health state. worker_registry.get_all() returns all workers including unhealthy ones, so when the first worker was temporarily down, proxy_get_request proxied to a dead worker and returned 500. This affected /v1/models, /get_server_info, and /get_model_info — all of which route through proxy_get_requestselect_first_worker.

Root-cause fix in the shared helper: filter for a healthy worker (is_healthy()) before returning; fall back to the existing error path when none are healthy.

Test Plan

New unit tests in routers::http::router::tests:

  • test_select_first_worker_skips_unhealthy — marks worker1 unhealthy, asserts worker2 is selected
  • test_select_first_worker_all_unhealthy — marks all unhealthy, asserts error

Test Result

running 3 tests
test routers::http::router::tests::test_select_first_worker_regular ... ok
test routers::http::router::tests::test_select_first_worker_skips_unhealthy ... ok
test routers::http::router::tests::test_select_first_worker_all_unhealthy ... ok

test result: ok. 3 passed; 0 failed

Built with cargo test --lib routers::http::router::tests::test_select_first_worker (rust:latest container, aarch64).

select_first_worker returned the first registered worker without checking
its health state, so /v1/models and other proxy GETs returned 500 when the
first worker was temporarily down. Filter for a healthy worker instead;
fall back to the existing error when none are healthy.

Fixes vllm-project#135

Signed-off-by: Binbin Zhang <binbin36520@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

/v1/models returns 500 when any worker is unhealthy

1 participant