tracking: split #946 (qwen35 TP sharding + batched decode + CUDA graph + eval tooling) into acceptance-bounded PRs

[Ma1oneZhang]

Background

PR #946 (feat(qwen35): shard GDR state per TP rank + batched eager/graph decode under TP, +2920/−316, 27 files) stacks four independently verifiable changes on one branch. Per reviewer request (comment), it needs to land as acceptance-bounded PRs. This issue tracks the split. The two open codex review comments on #946 are absorbed by the relevant splits below.

Splits

• 2b — TP correctness / memory: sharded linear-attention / GDR state per TP rank (weight-loader stitch/shard, per-rank stitched qkv/conv1d + row/col shards, state/buffers at local sizes, all-reduce after local linear out_proj, TP-local group-6 decode reroute). Acceptance: 27B TP2 fits on 2×48 GB (Phase-1 replicated state OOMed); 27B/9B HF logits + e2e gates pass on the rebased head.
• Batched eager TP decode: one batched run_decode_batch over all decode rows (step-scoped pointer table, single batched rank-0 select, per-row fan-out) replacing per-request bs=1 ticks. Acceptance: throughput A/B on its own (feat(qwen35): shard GDR state per TP rank + batched eager/graph decode under TP (+27B benchmark validation) #946 measured 16-concurrency aggregate 24.9 → 292.3 tok/s); gates unchanged.
• TP CUDA Graph: startup pre-capture sweep with warmup collectives per bucket message size, slot-indexed per-rank recurrent/conv state with scheduler-owned slot_idx + D2D compaction on retire, replay-only serving, 600 s watchdog. Gate: enable_cuda_graph && local_decode_group_is_compiled(tp); 27B TP2 (group-6) stays batched eager. Acceptance: graph A/B (feat(qwen35): shard GDR state per TP rank + batched eager/graph decode under TP (+27B benchmark validation) #946 measured 9B TP2 steady 706 → 767 tok/s, TPOT −8.8%); HF gates eager+graph PASS. Also absorbs the codex P2 on graph-slot reserve sizing (see below).
• Eval tooling + benchmark docs: scripts/eval_mc.py, scripts/eval_rerun_truncated.py, docs/benchmarks/qwen35-27b-tp2-knowledge-eval.md, tp-implementation records — after an exact-head runtime rerun of the benchmark numbers.

Open codex review comments absorbed by the split

1. P2 — pegainfer-qwen35/src/tp_executor.rs:1364: graph-slot preflight subtracts the reserve for bucket_for(requested_max_batch) before computing the downshifted capacity, so a memory-tight launch (e.g. default batch 64) can fail with effective capacity 0 although a smaller bucket would fit. Compute the reserve iteratively from the candidate effective batch. → split 3.
2. P2 — scripts/eval_mc.py:106: MMLU-Redux extraction is re.search(r'([ABCD])') over the whole completion, so a response starting with "Answer: B" scores as A. Prefer answer markers / standalone option tokens (mirror the SuperGPQA patterns in the same file). → split 4. Until this lands and numbers are rerun, the MMLU-Redux 94.09 figure on feat(qwen35): shard GDR state per TP rank + batched eager/graph decode under TP (+27B benchmark validation) #946 should not be cited externally.

#946 is superseded once all four splits merge.

[Ma1oneZhang]

Split PRs opened (all rebased onto origin/main @ 01f1e9a):

• Split 1 → feat(qwen35): shard linear-attention/GDR state per TP rank (split 1/4 of #946) #1003 (TP sharding; lib 106/106)
• Split 2 → perf(qwen35): batch eager decode rows under TP (split 2/4 of #946) #1004 (batched eager TP decode; lib 106/106)
• Split 3 → feat(qwen35): cuda graph decode under TP (split 3/4 of #946) #1005 (TP CUDA graph + effective-bucket reserve fix; lib 109/109)
• Split 4 → docs(qwen35): TP2 implementation record + knowledge-eval tooling (split 4/4 of #946) #1006 (docs + eval_mc.py MMLU-Redux extraction fix)

Both codex comments absorbed. Local gates on all branches: check/clippy -D warnings/fmt clean, lib tests green (7 GPU-gated skipped — no weights on build host). Remaining: GPU acceptance rerun on the rebased stack (27B TP2 HF/E2E, throughput A/B, graph A/B) before citing the original 2026-08-20 numbers.