fix(nemotron_3.5_super): pin a vllm-router build with prefix affinity fixes - #3080
Merged
Conversation
…ill affinity The wheel build pins the head of vllm-project/router#216, which fixes the decode-node hot spot (issue #197). Shipping #216 alone turns out to be worse than not shipping it for prefill-heavy benchmarks such as SWE-bench Pro. #216 makes the worker load counters honest, and that switches on a second latent bug: cache_aware decides whether to use prefix affinity from the fleet-wide load spread, so one hot worker discards affinity for every request -- including requests whose own worker is idle. Under P/D disaggregation that gate is open almost permanently, because prefill worker load counts queued requests as well as running ones. Routing degenerates to shortest-queue, already-cached prompts get recomputed, prefill saturates and decode starves behind it. Repin to the head of vllm-project/router#238, which carries #216 unchanged plus a fix applying the same load check per request, against the worker the request actually wants, rather than against the fleet maximum. Still fetched from vllm-project/router by bare SHA, exactly as the #216 pin already was. Signed-off-by: Sugam Devare <sdevare@nvidia.com>
sdevare-nv
force-pushed
the
sdd/vllm-router-prefill-affinity
branch
from
September 4, 2026 16:12
da804e4 to
82ac7d9
Compare
bxyu-nvidia
approved these changes
Sep 4, 2026
bxyu-nvidia
pushed a commit
that referenced
this pull request
Sep 4, 2026
#3107) The pin added in #3080 pointed at a commit that also carried per-role balance threshold flags (--prefill-balance-abs-threshold and friends). They have been dropped from vllm-project/router#238 to keep the PR to the bug fix, so this moves the pin to the new head. No behaviour change. The remaining commits are unchanged: #216 (health checker no longer resets in-flight worker load) plus the fix for the prefill-side regression #216 exposes, where cache_aware decided prefix affinity from the fleet-wide load spread instead of per request. Signed-off-by: Sugam Devare <sdevare@nvidia.com>
jkyi-nvidia
pushed a commit
that referenced
this pull request
Sep 11, 2026
… fixes (#3080) ## What does this PR do? Repins the `vllm-router` build used by the Super 3.5 eval container, from the head of [vllm-project/router#216](vllm-project/router#216) to the head of [#238](vllm-project/router#238). #216 fixes the decode-node hot spot (issue #197), but shipping it alone is worse than not shipping it for prefill-heavy benchmarks like SWE-bench Pro. #216 makes the worker load counters honest, and that switches on a second latent bug: `cache_aware` decides whether to use prefix affinity from the **fleet-wide** load spread, so one hot worker discards affinity for *every* request — including requests whose own worker is idle. Under P/D disaggregation that gate is open almost permanently, because prefill worker load counts queued requests as well as running ones. Routing degenerates to shortest-queue, already-cached prompts get recomputed, prefill saturates, and decode starves behind it. #238 carries #216 unchanged plus a fix that applies the same load check per request, against the worker the request actually wants, rather than against the fleet maximum. Still fetched from `vllm-project/router` by bare SHA, exactly as the #216 pin already was — a PR head is a ref in the upstream repo even when the branch lives on a contributor's fork. ## Testing Validated on SWE-bench Pro (4 prefill / 6 decode) against two baselines on the same checkpoint and vLLM config: the pre-#216 router, and #216 alone. With this pin the prefill prefix-cache hit rate and end-to-end throughput return to pre-#216 behaviour while keeping #216's decode hot-spot fix. Signed-off-by: Sugam Devare <sdevare@nvidia.com> Signed-off-by: Junkeun Yi <jkyi@nvidia.com>
jkyi-nvidia
pushed a commit
that referenced
this pull request
Sep 11, 2026
#3107) The pin added in #3080 pointed at a commit that also carried per-role balance threshold flags (--prefill-balance-abs-threshold and friends). They have been dropped from vllm-project/router#238 to keep the PR to the bug fix, so this moves the pin to the new head. No behaviour change. The remaining commits are unchanged: #216 (health checker no longer resets in-flight worker load) plus the fix for the prefill-side regression #216 exposes, where cache_aware decided prefix affinity from the fleet-wide load spread instead of per request. Signed-off-by: Sugam Devare <sdevare@nvidia.com> Signed-off-by: Junkeun Yi <jkyi@nvidia.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What does this PR do?
Repins the
vllm-routerbuild used by the Super 3.5 eval container, from the head ofvllm-project/router#216 to the head
of #238.
#216 fixes the decode-node hot spot (issue #197), but shipping it alone is worse than
not shipping it for prefill-heavy benchmarks like SWE-bench Pro.
#216 makes the worker load counters honest, and that switches on a second latent bug:
cache_awaredecides whether to use prefix affinity from the fleet-wide loadspread, so one hot worker discards affinity for every request — including requests
whose own worker is idle. Under P/D disaggregation that gate is open almost
permanently, because prefill worker load counts queued requests as well as running
ones. Routing degenerates to shortest-queue, already-cached prompts get recomputed,
prefill saturates, and decode starves behind it.
#238 carries #216 unchanged plus a fix that applies the same load check per request,
against the worker the request actually wants, rather than against the fleet maximum.
Still fetched from
vllm-project/routerby bare SHA, exactly as the #216 pin alreadywas — a PR head is a ref in the upstream repo even when the branch lives on a
contributor's fork.
Testing
Validated on SWE-bench Pro (4 prefill / 6 decode) against two baselines on the same
checkpoint and vLLM config: the pre-#216 router, and #216 alone. With this pin the
prefill prefix-cache hit rate and end-to-end throughput return to pre-#216 behaviour
while keeping #216's decode hot-spot fix.