Skip to content

fix(nemotron_3.5_super): pin a vllm-router build with prefix affinity fixes - #3080

Merged
bxyu-nvidia merged 1 commit into
mainfrom
sdd/vllm-router-prefill-affinity
Sep 4, 2026
Merged

fix(nemotron_3.5_super): pin a vllm-router build with prefix affinity fixes#3080
bxyu-nvidia merged 1 commit into
mainfrom
sdd/vllm-router-prefill-affinity

Conversation

@sdevare-nv

@sdevare-nv sdevare-nv commented Sep 4, 2026

Copy link
Copy Markdown
Contributor

What does this PR do?

Repins the vllm-router build used by the Super 3.5 eval container, from the head of
vllm-project/router#216 to the head
of #238.

#216 fixes the decode-node hot spot (issue #197), but shipping it alone is worse than
not shipping it for prefill-heavy benchmarks like SWE-bench Pro.

#216 makes the worker load counters honest, and that switches on a second latent bug:
cache_aware decides whether to use prefix affinity from the fleet-wide load
spread, so one hot worker discards affinity for every request — including requests
whose own worker is idle. Under P/D disaggregation that gate is open almost
permanently, because prefill worker load counts queued requests as well as running
ones. Routing degenerates to shortest-queue, already-cached prompts get recomputed,
prefill saturates, and decode starves behind it.

#238 carries #216 unchanged plus a fix that applies the same load check per request,
against the worker the request actually wants, rather than against the fleet maximum.

Still fetched from vllm-project/router by bare SHA, exactly as the #216 pin already
was — a PR head is a ref in the upstream repo even when the branch lives on a
contributor's fork.

Testing

Validated on SWE-bench Pro (4 prefill / 6 decode) against two baselines on the same
checkpoint and vLLM config: the pre-#216 router, and #216 alone. With this pin the
prefill prefix-cache hit rate and end-to-end throughput return to pre-#216 behaviour
while keeping #216's decode hot-spot fix.

…ill affinity

The wheel build pins the head of vllm-project/router#216, which fixes the
decode-node hot spot (issue #197). Shipping #216 alone turns out to be worse than
not shipping it for prefill-heavy benchmarks such as SWE-bench Pro.

#216 makes the worker load counters honest, and that switches on a second latent
bug: cache_aware decides whether to use prefix affinity from the fleet-wide load
spread, so one hot worker discards affinity for every request -- including requests
whose own worker is idle. Under P/D disaggregation that gate is open almost
permanently, because prefill worker load counts queued requests as well as running
ones. Routing degenerates to shortest-queue, already-cached prompts get recomputed,
prefill saturates and decode starves behind it.

Repin to the head of vllm-project/router#238, which carries #216 unchanged plus a
fix applying the same load check per request, against the worker the request
actually wants, rather than against the fleet maximum. Still fetched from
vllm-project/router by bare SHA, exactly as the #216 pin already was.

Signed-off-by: Sugam Devare <sdevare@nvidia.com>
@sdevare-nv sdevare-nv changed the title fix(nemotron_3.5_super): pin a vllm-router build that also fixes pref… fix(nemotron_3.5_super): pin a vllm-router build with prefix affinity fixes Sep 4, 2026
@sdevare-nv
sdevare-nv force-pushed the sdd/vllm-router-prefill-affinity branch from da804e4 to 82ac7d9 Compare September 4, 2026 16:12
@bxyu-nvidia
bxyu-nvidia merged commit 2448579 into main Sep 4, 2026
60 checks passed
@bxyu-nvidia
bxyu-nvidia deleted the sdd/vllm-router-prefill-affinity branch September 4, 2026 19:23
bxyu-nvidia pushed a commit that referenced this pull request Sep 4, 2026
#3107)

The pin added in #3080 pointed at a commit that also carried per-role
balance threshold flags (--prefill-balance-abs-threshold and friends).
They have been dropped from vllm-project/router#238 to keep the PR to
the bug fix, so this moves the pin to the new head.

No behaviour change. The remaining commits are unchanged: #216 (health
checker no longer resets in-flight worker load) plus the fix for the
prefill-side regression #216 exposes, where cache_aware decided prefix
affinity from the fleet-wide load spread instead of per request.

Signed-off-by: Sugam Devare <sdevare@nvidia.com>
jkyi-nvidia pushed a commit that referenced this pull request Sep 11, 2026
… fixes (#3080)

## What does this PR do?

Repins the `vllm-router` build used by the Super 3.5 eval container,
from the head of

[vllm-project/router#216](vllm-project/router#216)
to the head
of [#238](vllm-project/router#238).

#216 fixes the decode-node hot spot (issue #197), but shipping it alone
is worse than
not shipping it for prefill-heavy benchmarks like SWE-bench Pro.

#216 makes the worker load counters honest, and that switches on a
second latent bug:
`cache_aware` decides whether to use prefix affinity from the
**fleet-wide** load
spread, so one hot worker discards affinity for *every* request —
including requests
whose own worker is idle. Under P/D disaggregation that gate is open
almost
permanently, because prefill worker load counts queued requests as well
as running
ones. Routing degenerates to shortest-queue, already-cached prompts get
recomputed,
prefill saturates, and decode starves behind it.

#238 carries #216 unchanged plus a fix that applies the same load check
per request,
against the worker the request actually wants, rather than against the
fleet maximum.

Still fetched from `vllm-project/router` by bare SHA, exactly as the
#216 pin already
was — a PR head is a ref in the upstream repo even when the branch lives
on a
contributor's fork.

## Testing

Validated on SWE-bench Pro (4 prefill / 6 decode) against two baselines
on the same
checkpoint and vLLM config: the pre-#216 router, and #216 alone. With
this pin the
prefill prefix-cache hit rate and end-to-end throughput return to
pre-#216 behaviour
while keeping #216's decode hot-spot fix.

Signed-off-by: Sugam Devare <sdevare@nvidia.com>
Signed-off-by: Junkeun Yi <jkyi@nvidia.com>
jkyi-nvidia pushed a commit that referenced this pull request Sep 11, 2026
#3107)

The pin added in #3080 pointed at a commit that also carried per-role
balance threshold flags (--prefill-balance-abs-threshold and friends).
They have been dropped from vllm-project/router#238 to keep the PR to
the bug fix, so this moves the pin to the new head.

No behaviour change. The remaining commits are unchanged: #216 (health
checker no longer resets in-flight worker load) plus the fix for the
prefill-side regression #216 exposes, where cache_aware decided prefix
affinity from the fleet-wide load spread instead of per request.

Signed-off-by: Sugam Devare <sdevare@nvidia.com>
Signed-off-by: Junkeun Yi <jkyi@nvidia.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants