qwen35: add adaptive decode-priority scheduling for mixed prefill/decode load

[CAICAIIs]

Summary

Qwen3.5 currently uses a fixed scheduling rule when active decode requests and pending prefill work coexist. This can protect throughput in some cases, but it can also create active-decode stalls under mixed workloads where long cold prefills arrive while existing requests are decoding.

This issue tracks an adaptive scheduler policy for Qwen3.5 that can reduce mixed prefill/decode ITL stalls without making the default serving path pay a broad TTFT or throughput regression.

Motivation

The current fixed policy is too coarse for a default latency policy:

• Always prioritizing prefill can stall active decode behind long prompt work.
• Always overlapping or shrinking prefill can improve ITL tail, but may regress TTFT and/or throughput.
• Static request fields such as max_tokens are not enough to decide whether a request should trigger decode protection.

A default policy should react to runtime state, not just a single CLI mode or fixed threshold.

Evidence

On a single RTX 5090 Qwen3.5 serving setup, opt-in stream overlap showed that active-decode stalls can be reduced, but it also exposed why the behavior should not simply become the default:

• Mixed-load ITL p99 improved substantially when overlap was enabled.
• Standard HTTP serving cells remained sensitive to TTFT and throughput tradeoffs.
• A max_tokens >= 1024 gate preserved short-output cells better, but still regressed median TTFT on a long-output concurrency cell.
• Active-only smaller prefill chunks improved some mixed-load tail metrics, but regressed c16 / QPS serving throughput and TTFT.

These results suggest the bottleneck is schedulable, but the current policies are not yet a safe default.

Proposed Direction

Add an adaptive Qwen3.5 scheduler policy that decides per scheduler tick how to balance decode and prefill work. Candidate signals include:

• active decode batch size
• pending prefill token count
• pending request queue age
• active request generated tokens / remaining budget
• recent ITL stall history
• prefill chunk size and estimated stall cost
• starvation/fairness limits for pending prefill

The policy should be able to choose among existing execution options, such as:

• normal prefill / decode / unified execution
• smaller prefill chunks while decode is active
• decode-priority ticks
• opt-in stream overlap as an execution mechanism when the runtime state justifies it

Acceptance Criteria

A default auto policy should meet all of the following before becoming the default:

• No material regression on standard HTTP serving cells such as 1024/256 c1 and c16.
• No material regression on QPS pressure cells such as 1024/128 QPS 8/12/16.
• Clear mixed-load ITL tail improvement when long cold prefills arrive during active decode.
• Long-output concurrency cells do not show a large TTFT regression.
• Failed, timeout, OOM, unsupported, and zero-output cases remain explicitly recorded.
• Output lengths and output sanity/hash checks are retained for every benchmark cell.
• The policy remains disableable with an explicit off mode.

Non-goals

• This issue does not claim vLLM parity.
• This issue does not make stream overlap the default by itself.
• This issue does not require kernel profiling unless scheduler-level evidence shows GPU step time is the dominant remaining bottleneck.
• This issue does not change prefix-cache behavior.

[xiaguan]

chunk prefill 不能缓解这个问题吗？

[CAICAIIs]

chunk prefill 不能缓解这个问题吗？

之前的策略太固定了，新请求优先做 prefill，老请求每轮 decode 一个 token，就会出现老请求正在 decode 时，新来请求要执行 prefill 的场景。举个 bad case 就是：用户正在输出 token 时突然插入长 prompt 的 prefill 任务，会拉长 decode 间隔。
即便引入 chunk prefill，原有调度策略没变，例如 8000 token 的 prompt 会按固定预算切分，每块 1024 token 循环执行；按旧策略，每一段 1024 的 prefill chunk 依旧排在 decode 任务前面执行。
这次主要是去新增调度策略，让 scheduler 实时判断老请求是否阻塞、新 prefill 请求长度与等待时长，根据实际去动态调整 chunk 大小和任务执行顺序

[bbirdxr]

@CAICAIIs Hi, I’ve completed the measurements for #470. Would it be possible for me to participate in this issue?

[xiaguan]

@bbirdxr Thanks for offering to continue. Please take the independent validation track for this issue and coordinate with @CAICAIIs, who already owns the policy implementation in PR #730. Please do not start a second scheduler implementation.

Validation scope:

• retain the ITL_STEP overlap validity gate from qwen35: measure mixed-load ITL after scheduler-level chunked prefill #470 and derive stall buckets from step timestamps rather than the broad [submit, last-token] request window;
• run the missing 1024/128 QPS 8/12/16 acceptance cells;
• add long-output concurrency coverage;
• cover more than the current bg_concurrency=4 mixed cell, including wider active-decode batches;
• report p50/p99/max, TTFT, throughput, completions/failures, output lengths/hashes, and saturated cells without hiding regressions.

PR #730 currently has neutral 1024/256 c1/c16 results and one narrow mixed cell where p99 improves but max regresses from 65.2 ms to 196.6 ms. That is useful evidence for continued policy development, but it is not enough to make Auto the default. The default flip remains blocked until the issue's retained acceptance matrix is complete; in the meantime the fixed behavior should remain the default and the new policy should be opt-in.

I have closed #470 as completed by merged PR #693, so #727 is now the single place for this policy decision and its remaining validation.

[CAICAIIs]

@CAICAIIs Hi, I’ve completed the measurements for #470. Would it be possible for me to participate in this issue?

@bbirdxr Yes, please feel free to continue with the validation track.

PR #730 is merged, but it only lands the policy plumbing with off as the default and auto as opt-in. The scheduler policy is still not mature enough for a default flip. The best next step is to complete the validation matrix @xiaguan listed, without starting a second scheduler implementation. Once the data shows a clear SLA target, we can tune the policy on top of the #730 path.

[bbirdxr]

Yes, I will continue with these validation works.

[bbirdxr]

Hi, here is my measurement.

1. Standard and long-output: neutral

Policy	Cell	TTFT p50 ms	steady TPOT p50/p99 ms	req tok/s	out len	hash0
off	1024/256 c1	66.261	10.971 / 11.157	89.35	256	0827a7035c7b7a89
auto	1024/256 c1	66.216	10.965 / 11.147	89.41	256	0827a7035c7b7a89
off	1024/256 c16	688.500	15.232 / 80.632	50.98	256	0827a7035c7b7a89
auto	1024/256 c16	691.312	15.233 / 80.678	50.95	256	0827a7035c7b7a89
off	1024/2048 c8	379.200	12.982 / 13.691	75.41	2048	dc6c576de0d38289
auto	1024/2048 c8	379.555	12.933 / 13.606	75.87	2048	dc6c576de0d38289

(auto−off)/off: TTFT ≤ +0.41%, TPOT p50 ≤ +0.01%, TPOT p99 ≤ +0.06%. Identical hash0, so the fixed-chunk path is bit-for-bit unchanged.

2. Mixed-load: #470 gate retained

Stall buckets come from scheduler step timestamps via OPENINFER_ITL_DEBUG and scripts/itl_step_agg.py. A stall step is prefill_tok > 0 AND decode_n > 0; the gate requires decode_n == bg_concurrency.

Cell family	off valid?	auto valid?
mixed_bg{8,16}_mb{16,32}_p{4096,8192}_q{0p5,1p0} (16 cells)	yes	yes
mixed_negctl_bg8_mb8 (max_batch == bg)	no (no decode_n=8 stall step; starvation warning)	no

Representative cell mixed_bg8_mb16_p4096_q0p5:

Policy	stall p50/p99/max ms	steady decode p50/p99 ms	stall steps @ decode_n=8
off	79.58 / 85.55 / 88.45	12.31 / 12.87	40
auto	48.48 / 83.75 / 84.71	12.34 / 12.86	40

3. Correction #1 — that p50 drop is not a win

I previously flagged the 79.58 → 48.48 ms stall p50 as possible evidence for auto. It is not. Splitting stall steps by forwarded chunk shape shows what actually happened:

Forwarded shape	off	auto
prefill_tok=512 reqs=1	5 steps @ 47.70 ms	47 steps @ 47.20 ms
prefill_tok=1024 reqs=1	40 steps @ 79.06 ms	40 steps @ 78.78 ms
prefill_tok=1024 reqs=2	21 steps @ 80.22 ms	none
all stall steps	66 (avg 77.33 ms)	87 (avg 62.03 ms)

The FIFO-front cap suppresses co-packing: the 21 co-packed off steps become 42 single-request 512-token steps under auto (5 + 42 = 47, 66 + 21 = 87 both close), while the 40 injection chunks are identical on both — the policy never
touches the injection path in this cell. So p50 moved because the distribution was split, not because anything ran faster. Total stall time rises from 5104 ms to 5397 ms (+5.7%), since every split step pays the ~12 ms per-step fixed cost again. openinfer-qwen3's own DEFAULT_MAX_PREFILL_TOKENS comment already records this from the other direction: "512 chunks no longer amortize the per-step fixed cost, so prefill falls behind arrivals and TTFT queues up."

Practical consequence: do not use a stall-percentile delta alone as evidence for this policy. It moves when the step-size distribution changes, in either direction, with no change in total work.

4. HTTP 1024/128 QPS 8/12/16 — regression is real

Policy	qps	completed	req/s	out tok/s	TTFT p50/p99 ms	TPOT p50/p99 ms	ITL p99 ms
off	8	480	5.25	672.0	14423 / 29891	23.15 / 23.38	82.60
auto	8	480	5.03	644.4	16326 / 33057	22.14 / 23.37	82.97
off	12	720	5.25	671.4	36798 / 75068	23.27 / 23.45	82.81
auto	12	720	5.07	649.6	38705 / 79656	22.15 / 23.30	83.00
off	16	960	5.28	675.3	59217 / 119843	23.19 / 23.50	82.69
auto	16	960	4.98	637.6	63750 / 128829	19.63 / 23.07	82.63

out tok/s: −4.1% / −3.2% / −5.6%. ITL p99 flat. TPOT p50 better on auto. 0 failures, completions = prompts.

It reproduces. A fresh qps-16 pair gives off 674.6 / auto 637.8 against the original 675.3 / 637.6. Two consecutive same-policy off runs differ by only 0.15%, so run-to-run noise on this harness is far below the effect.

5. Correction #2 — the mechanism, measured

I first guessed a final-chunk trim, then estimated a ~40% tick hit rate from the window share. Both were wrong in detail. Step accounting settles it — qps 16 with OPENINFER_ITL_DEBUG=1:

Step accounting @ qps 16	off	auto
total ITL_STEP	7698	8583
decode-only steps	6738	7623
prefill-executing steps	960	960
co-packed steps (prefill_reqs=2)	4	0
wall time	182.2 s	192.7 s

Prefill work is identical — 960 prefill-executing steps either way. auto simply inserts 885 extra decode-only steps. At the measured 15.05 ms steady decode step that is ~10.6 s, against an observed +10.5 s (182.2 → 192.7 s). That accounts for the entire throughput delta. The FIFO-front cap contributes nothing here: INPUT_LEN=1024 equals the 1024 base budget, leaving 4 co-packed steps to suppress. TPOT p50 23.24 → 19.36 ms is the same mechanism seen from the decode side.

Cross-check at 1024/512 (qps 16, SECONDS_PER_RUN=20, 320 prompts):

1024/512 @ qps 16	off	auto	delta
out tok/s	909.4	891.9	−1.9%
decode-only steps	9937	10222	+285
TPOT p50 ms	17.48	16.50	−5.6%

The extra decode-only steps scale with completed requests, not wall time: 285/885 = 0.32 against 320/960 = 0.33. The tick fires ~0.9 times per completed request in both shapes — well under the 4-token window depth, because it also needs a prefill in flight to have anything to defer. So a 4× longer output dilutes a fixed per-request cost and the gap shrinks accordingly. The price of auto is predictable from output_len, which also means the 1024/128 shape is close to its worst case.

[github-actions]

This issue has been inactive for 14 days. It will be closed after another 30 days unless there is new activity.