diff --git a/benchmarks/nemotron_3.5_super/README.md b/benchmarks/nemotron_3.5_super/README.md index a6cc07b2a5..0ebea8f2e7 100644 --- a/benchmarks/nemotron_3.5_super/README.md +++ b/benchmarks/nemotron_3.5_super/README.md @@ -40,6 +40,24 @@ worker every 10 cycles regardless of load. - bug: https://github.com/vllm-project/router/issues/197 - fix: https://github.com/vllm-project/router/pull/216 (unmerged upstream) +#216 on its own is not enough, and for prefill-heavy benchmarks it is worse than not +applying it. It makes the worker load counters honest, which switches on a second +latent bug: `cache_aware` decides whether to use prefix affinity from the +*fleet-wide* load spread, so one hot worker discards affinity for every request -- +including requests whose own worker is idle. Under P/D that gate is open almost +permanently, because prefill worker load counts queued requests as well as running +ones. Routing degenerates to shortest-queue, already-cached prompts get recomputed, +prefill saturates and decode starves behind it. + +The pin therefore points at a branch carrying #216 plus a fix that applies the same +load check per request, against the worker that request wants: + +- prefill fix: https://github.com/vllm-project/router/pull/238 + +Both are plain commit SHAs fetched from `vllm-project/router`; a PR head is a ref +there even when the branch lives on a contributor's fork. Repin to a released commit +once these land upstream. + Build the wheel once with `build_vllm_router_wheel.sh`, then pass it to the container build. The wheel is built inside the eval base image, so its extension module matches the Python that runs `vllm-router` at eval time: diff --git a/benchmarks/nemotron_3.5_super/build_vllm_router_wheel.sh b/benchmarks/nemotron_3.5_super/build_vllm_router_wheel.sh index d19f7d0c70..7067be2f55 100755 --- a/benchmarks/nemotron_3.5_super/build_vllm_router_wheel.sh +++ b/benchmarks/nemotron_3.5_super/build_vllm_router_wheel.sh @@ -30,10 +30,11 @@ set -euo pipefail CONTAINER=$CONTAINER OUTPUT_DIR=${OUTPUT_DIR:-$(pwd)/results/vllm_router} VLLM_ROUTER_GIT_URL=${VLLM_ROUTER_GIT_URL:-https://github.com/vllm-project/router} -# Head of vllm-project/router#216 ("[Bugfix]: Fix health checker to preserve -# worker load during checks"). Fetching a bare SHA works because GitHub serves -# any commit reachable from a ref, and PR heads are refs. -VLLM_ROUTER_COMMIT=${VLLM_ROUTER_COMMIT:-9e6fce282a877c65185468692c6ba8a483409d9b} +# Head of vllm-project/router#238, which carries #216 ("[Bugfix]: Fix health checker +# to preserve worker load during checks") unchanged plus a fix for the prefill-side +# regression #216 exposes; see the README. Fetching a bare SHA works because GitHub +# serves any commit reachable from a ref, and PR heads are refs. +VLLM_ROUTER_COMMIT=${VLLM_ROUTER_COMMIT:-5b7ce08ea8cd27123cf6c23313cd92d3ba0da70e} RUST_TOOLCHAIN=${RUST_TOOLCHAIN:-1.95.0} # Load-accounting unit tests from the fix. Compiling the test binary roughly # doubles the job, so it is opt-out.