feat(qwen35): keep the auto prefill budget running under stream overlap - #1034
Conversation
With 'auto + stream' the finishing-window full deferral (a decode-priority
tick when an active request has <= 4 tokens left) no longer buys decode
latency: the overlapped chunk already runs off the decode step, so the
deferral only delays prefill. Skip it when overlap is enabled and keep the
capped FIFO-front budget.
A100-40GB, 1024-token prompts vs the merged combination ('70a600b7'+pegainfer-project#1033):
| QPS16 1024/128 | TPOT | ITL p99 | TTFT | out tok/s |
| --- | --- | --- | --- | --- |
| combo (window kept) | 20.81 | 41.71 | 1828 | 873 |
| combo, overlap-aware | 23.77 | 41.65 | 1264 | 992 |
| vLLM 0.27.0 | 23.60 | 93.43 | 218 | 1404 |
c8/c16 unchanged within run noise (p99 34.5/36.4 ms). The redundant QPS16
TPOT win (already parity with vLLM) is traded for 31% TTFT and 14%
throughput at identical tail. The remaining TTFT/throughput gap to vLLM is
total-throughput-bound (kernel-side), not schedulable.
Evidence: lib tests 98 passed (incl. the new
adaptive_prefill_budget_keeps_prefill_running_under_stream_overlap),
hf_golden_gate TP1 2/2 + TP2 (--ignored) 2/2, e2e_scheduler passed; HTTP
cells above with zero failed requests (raw JSONs under ~/ab_cfg_combo2/ on
the bench host).
Signed-off-by: CAICAIIs <3360776475@qq.com>
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 44ad2ee04d
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
Review follow-up on pegainfer-project#1034: the stream-overlap finishing-window change and its A/B evidence update this doc's guidance, so the Last touched stamp must move with it (AGENTS.md docs rule). Signed-off-by: CAICAIIs <3360776475@qq.com>
Review pass on pegainfer-project#1034: the Single-backend overlap probe ran three times per step (budget choice, ITL plan kind, Unified dispatch) even though the backend is fixed for the loop's lifetime. Hoist one binding, and mark the policy-doc measurement as a single run. Signed-off-by: CAICAIIs <3360776475@qq.com>
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 4f9d087e0f
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
| > longer stalls decode), which reclaims most of its open-loop cost: QPS16 TTFT | ||
| > `1828 → 1264 ms`, output throughput `873 → 992 tok/s`, ITL p99 unchanged; | ||
| > TPOT returns to vLLM parity (`23.8 vs 23.6 ms`). |
There was a problem hiding this comment.
The new 1828 → 1264 / 873 → 992 figures were measured from fb16fc15 plus this patch according to the commit's benchmark contract, but here they remain under the source binding on lines 12–13 that attributes the combination evidence to 70a600b7 + this change. Those revisions straddle earlier overlap-policy work, so this prevents a reproducible same-context A/B and can cause readers to attribute the result to the wrong build; record the exact source for the new run separately.
AGENTS.md reference: AGENTS.md:L121-L123
Useful? React with 👍 / 👎.
| 1024, | ||
| "with stream overlap the finishing window keeps prefill on the prefill stream instead of deferring it" | ||
| ); | ||
| assert!( |
There was a problem hiding this comment.
no blocking but the first assertion correctly exercises the changed branch: a decoder has four tokens left, overlap is enabled, and the budget remains 1024 instead of returning zero. The final assertion supplies a separate empty queue to build_next_plan(), so it does not use that budget or establish the overlap behaviour, its “zero prefill budget” explanation contradicts this case's expected result.
Address review on pegainfer-project#1034: the finishing-window deferral reclaim numbers (TTFT 1828 -> 1264 ms, output 873 -> 992 tok/s) were measured on fb16fc1 + this change, so record that revision separately from the 70a600b-based combination table in unified-prefill-overlap.md. Rework the stream-overlap budget test's tail assertion per review: feed the same non-empty prefill queue through build_next_plan and assert the tick stays a unified prefill+decode plan, instead of asserting a decode-only tick on a separate empty queue whose message contradicted the case's kept-budget expectation. adaptive-scheduler-policy.md's Last touched already reads 2026-09 (8919e81). Signed-off-by: CAICAIIs <3360776475@qq.com>
2 participants
Summary
Make the
autoscheduler policy's finishing-window deferral overlap-aware: with--decode-overlap stream, the window no longer defers the whole prefill budget, because the overlapped chunk already runs off the decode step — the deferral only delayed prefill.Background
#1033 allowed
auto+stream. The follow-up measurement showed the combination's one open-loop cost — QPS16 TTFT1828 vs 1484 ms(stream alone) — comes almost entirely from this window: with many short decoders, some active request is within the 4-token finish window on most steps, and each such step runs decode-only while the prefill waits.Change
choose_prefill_budgettakes the overlap mode and skips the finishing-window full deferral when stream overlap is enabled; the capped FIFO-front budget (min(remaining, --max-prefill-tokens)) applies instead.offoverlap) semantics are unchanged; new unit testadaptive_prefill_budget_keeps_prefill_running_under_stream_overlappins the overlap branch.adaptive-scheduler-policy.md,unified-prefill-overlap.md).Evidence (1x A100-40GB, upstream/main
fb16fc15+ this change, vLLM 0.27.0 baseline)1024-token prompts, greedy,
vllm bench serve, zero failed requests; single run per cell:c8/c16 unchanged within run noise (ITL p99
34.5 / 36.4 ms). The redundant QPS16 TPOT win (already parity with vLLM) is traded for 31% TTFT and 14% throughput at an identical tail. The remaining TTFT/throughput gap to vLLM is total-throughput-bound (kernel-side batch decode slope), not schedulable.Correctness: release lib tests 98 passed (including the new overlap-branch unit test),
hf_golden_gateTP1 2/2 and TP2 (--ignored) 2/2,e2e_schedulerpassed.Claim boundary
Both knobs remain opt-in (
offdefaults);streamkeeps the--max-batch <= 32cap and single-GPU-only restriction. Single-run numbers on one GPU — not a parity or production-readiness claim.