pegainfer-project · Ma1oneZhang · Aug 20, 2026 · Aug 20, 2026 · Aug 20, 2026 · Aug 20, 2026
diff --git a/.gitignore b/.gitignore
@@ -10,3 +10,4 @@ __pycache__/
 /docs/private/
 /profile/
 .claude/scheduled_tasks.lock
+/results/
diff --git a/docs/benchmarks/qwen35-27b-tp2-knowledge-eval.md b/docs/benchmarks/qwen35-27b-tp2-knowledge-eval.md
@@ -0,0 +1,51 @@
+# Qwen3.5-27B TP2 Knowledge Benchmarks (vs Official Scores)
+
+> TL;DR: Qwen3.5-27B on pegainfer TP2 (2× RTX 4090, batched eager decode) runs the knowledge benchmarks at C-Eval 88.11 (official 90.5) and MMLU-Redux 94.09 (official 93.2), both inside the normal cross-harness band; MMLU-Pro / SuperGPQA only completed sampled smokes due to runtime cost, no final scores (see below). No TP-induced accuracy issue in the model numerics.
+>
+> Note: scores were measured on the pre-rebase in-house Phase 1/2a line; after the rebase onto #870 the logits golden gates pass identically on both sides, so the numerics carry over — but rerun on this PR branch before formally citing parity.
+
+## Environment
+
+- GPU: 2× RTX 4090 (48 GB variant), `--tp-size 2 --cuda-graph false` (TP + CUDA Graph still fail-closed)
+- Model: Qwen/Qwen3.5-27B `fc05daec`, BF16, served-model-name `qwen35-27b-tp2`
+- Sampling: temperature=0.0, top_p=1.0, chat completions (thinking mode, i.e. the template default)
+- Evaluator: `scripts/eval_mc.py` (in-house, uniform `/v1/chat/completions` at concurrency 48), recipes replicated item-by-item from the official harnesses:
+  - **C-Eval** = OpenCompass `ceval_gen`: 52 subjects, full val split (1346 items), 5-shot from the dev split, `"答案: "` continuation, first-capital-letter extraction
+  - **MMLU-Redux** = lm-eval `mmlu_redux_generative`: `fxmarty/mmlu-redux-2.0-ok`, 57 subjects, full test split (5330 items), 0-shot, first `[ABCD]` extraction
+  - **MMLU-Pro** = lm-eval `mmlu_pro`: TIGER-Lab/MMLU-Pro test, 5-shot CoT from the validation split, `answer is (X)` extraction
+  - **SuperGPQA** = OpenCompass `supergpqa_gen`: `m-a-p/SuperGPQA` train (26529 items), 0-shot, two-tier "Answer: X" letter/content extraction
+- Launch command: `./target/release/pegainfer --model-path <27B> --served-model-name qwen35-27b-tp2 --tp-size 2 --cuda-graph false --port 18082` (NCCL runtime setup in `docs/playbooks/developer-onboarding.md`)
+
+## Results (as of 2026-08-20; runs terminated early as needed)
+
+| Benchmark | Official | Measured | n | Protocol | Δ verdict |
+|---|---|---|---|---|---|
+| MMLU-Redux | 93.2 | **94.09** | 5330 full | 8192 cap + truncated-rerun merge (32 rerun, 2 still truncated) | **in band** |
+| C-Eval | 90.5 | **88.11** | 1346 full | 8192 cap + truncated-rerun merge (48 rerun, 0 residual) | **band edge** (-2.4pp, CI95 ±1.7pp) |
+| MMLU-Pro | 86.1 | — | stopped at 600/2000 | sampled n=2000 (cap 24576) manually stopped at 30%; 100-item smoke truncated 51% at cap 4096 | no final score |
+| SuperGPQA | 65.6 | — | no formal run | 100-item smoke: 43 completable items, 27/43 ≈ 63% correct against gold | no final score |
+
+## Key Observations
+
+- **No TP accuracy issue**: the 27B TP2 HF logits golden gates are all green; on the non-truncated C-Eval subset (1290/1346) accuracy is 90.2% ≈ official 90.5. The C-Eval gap comes entirely from the hardest items cut off by the thinking-length cap, not from model miscomputation.
+- **MMLU-Redux slightly above official** (+0.9pp): in band; the prompt/extraction/numerics chain is sound.
+- **Thinking length is the dominant system variable**: on C-Eval ~4% of items need >8192 tokens with a thinking model; on MMLU-Pro over a third of items do not finish within 4096. The official harnesses' max_tokens is unknown (presumed ≥32k); this eval approximates with an 8192 first pass plus a 32768 rerun merge. ±1–2pp across harnesses is normal.
+- **Throughput precondition**: this eval is feasible only because of the Step 3 batched eager TP decode fix (before it, 16 concurrent requests aggregated ~25 tok/s and full runs were infeasible; after it, ~450 tok/s at 48 concurrent).
+
+## Reproduction Commands
+
+```bash
+# server (see above); eval:
+.venv/bin/python -u scripts/eval_mc.py ceval --max-tokens 8192 --concurrency 48 --out-dir results/qwen35-27b-tp2-eval
+.venv/bin/python -u scripts/eval_mc.py mmlu_redux --max-tokens 8192 --concurrency 48 --out-dir results/qwen35-27b-tp2-eval
+# rerun-and-merge truncated samples:
+.venv/bin/python -u scripts/eval_rerun_truncated.py ceval --max-tokens 32768 --concurrency 16
+# sampling (--sample stratifies by subject proportionally, seed 1337; at n=2000 the CI95 half-width is ±1.3pp):
+.venv/bin/python -u scripts/eval_mc.py mmlu_pro --sample 2000 --max-tokens 24576 --concurrency 48
+.venv/bin/python -u scripts/eval_mc.py supergpqa --sample 2000 --max-tokens 24576 --concurrency 48
+```
+
+## Next Steps
+
+- Complete the full sampled runs of MMLU-Pro / SuperGPQA (2000 items each, ~10h total estimated at ~450 tok/s).
+- To converge C-Eval to within official ±1pp: rerun the full set with a larger thinking budget (no merge), and confirm whether the Qwen official harness prompt template matches this eval's.
diff --git a/docs/index.md b/docs/index.md
@@ -53,8 +53,8 @@ Organized by domain (model line / subsystem / playbook / lesson) instead of by l
 | `models/qwen35/accuracy.md` | Qwen3.5 HF bf16 logits goldens, size-keyed (0.8b/2b/4b/9b/27b all committed), through `past_key_values`: short replay covers sequential graph, bucket-straddling batched graph, and slot-compaction; long replay covers 4097/8192-token prompts; full GSM8K 8-shot now matches the HF baseline within 0.15 percentage points. |
 | `models/qwen35/model-crate.md` | `pegainfer-qwen35` owns Qwen3.5 model/scheduler/recurrent ops/tests/benches; feature-gated behind `qwen35` (Triton AOT is the only Python build dependency); root loads it through `EngineHandle`. Build/check/clippy, root bench sanity check, historical Qwen3.5 e2e, and scheduler e2e records live here. |
 | `models/qwen35/batched-step-tail.md` | Qwen3.5 issue #353 implementation record: final prefill tail is batched, decode/unified sample from batched logits, host full-vocab copies are logprobs-only, HF + scheduler e2e pass, and final serving A/B supports only the first-token/short-output TTFT claim. |
-| `models/qwen35/tp-design.md` | Qwen3.5 TP design: Phase 1 is eager dense TP on Qwen3's controller/worker runtime; validate TP2 first, fail closed for indivisible degrees and TP+CUDA Graph, shard dense full-attention/MLP, and leave sharded linear/GDR state to follow-up. |
-| `models/qwen35/tp-implementation.md` | Qwen3.5 TP Phase 1 and P2A are complete: TP2 has start-gated eager unified prefill+decode, strict ID-aligned artifacts, fail-closed lifecycle recovery, and pre-load ordinal validation; P2B GDR state sharding is next. |
+| `models/qwen35/tp-design.md` | Qwen3.5 TP design: Phase 1 eager dense TP on Qwen3's controller/worker runtime, P2a mixed-step protocol, P2b rank-local GDR sharding, P2c CUDA Graph under TP gated on the compiled decode GQA group (27B group-6 stays eager). |
+| `models/qwen35/tp-implementation.md` | Qwen3.5 TP landed through P2c on #870 (2026-08, 2× RTX 4090): Phase 1/P2A lifecycle and ID contracts kept; GDR state sharded per rank (27B TP2 fits 48 GB pairs); batched eager decode (27B: 292 tok/s ×16); TP decode CUDA Graphs for 4B/9B (9B: 767 vs 706 tok/s ×16 eager). 9B/27B TP2 HF + e2e gates pass. |
 | `models/qwen35/mixed-load-itl-470.md` | Issue #470: full cold `--max-batch 8/bg=4` matrix on RTX 4090 (24/24 valid) + starvation negative control. Qwen3.5 is not immune; chunking bounds max/per-step stall but raises p99 at low QPS (~14→~80–92ms) and pulls p99/max back from the prefill wall to the chunk wall at high load; `qps·prefill_s≳1` is a throughput wall (chunking can't fix it, and ON's +15% TTFT can trip it earlier). The old "p99 immunity" was a slot-starvation artifact. |
 | `models/qwen35/adaptive-scheduler-policy.md` | Issue #727 adaptive scheduler policy record: default `off`, opt-in `auto`, hard `--max-prefill-tokens` cap, TP `auto` rejection, and pre-review whole-prefill benchmark tradeoff retained as non-default evidence. |
 | `models/qwen35/unified-prefill-overlap.md` | Issue #715 implementation record: opt-in single-GPU shared-SM overlap keeps one prefill chunk in flight while active decode continues; default serial policy and unsupported-combination guards remain explicit. |
@@ -232,6 +232,7 @@ Organized by domain (model line / subsystem / playbook / lesson) instead of by l
 | `benchmarks/bs1-4k64-vllm-pegainfer.md` | RTX 5090 single-concurrency probe: `input_len=4096`, `output_len=64`, no vLLM prefix cache. PegaInfer TTFT median `177ms` vs vLLM `198ms`; TPOT median `6.47ms` vs `6.36ms`; corrected output throughput `+6%` for PegaInfer. |
 | `benchmarks/mixed-load-itl.md` | Qwen3-4B + Qwen3.5 mixed-load ITL (#244, #375): chunking-off sweeps via `bench_serving mixed`. Both freeze active decode for the full prefill. Qwen3 p99 blows up with prompt/QPS; the old Qwen3.5 “p99-immune” table is a **measurement artifact** (primary: hardcoded `max_batch=4` slot starvation — see #470 / `models/qwen35/mixed-load-itl-470.md`; secondary: short `bg_output_len`). Prefix reuse defeats it on Qwen3. |
 | `benchmarks/accuracy-eval-results.md` | Phase 1 GSM8K: Qwen3-4B PASS (pegainfer 85.37% vs HF 85.82%, delta -0.45 pp). Qwen3.5-4B historical FAIL recovered by #250 (strict 79.38%, flexible 79.30% vs HF 79.45%). |
+| `benchmarks/qwen35-27b-tp2-knowledge-eval.md` | Qwen3.5-27B TP2 (2× RTX 4090, batched eager decode) knowledge benchmarks vs official: MMLU-Redux 94.09 vs 93.2 (full 5330), C-Eval 88.11 vs 90.5 (full 1346, thinking-cap truncation rerun-merged) — both in cross-harness band; MMLU-Pro/SuperGPQA partial smokes only. |
 | `benchmarks/qwen3-8b-pd-vs-mix-h200.md` | Qwen3-8B 多轮负载三方 A/B（2×H200）：P/D 1P+1D vs mixed×2（会话亲和 LB）vs mixed×1。吞吐持平（47.8k vs 47.0k tok/s），P/D 赢在 decode 稳定性（TPOT p99 10.08 vs 12.77ms，turn2+ TTFT 恒定 ~107ms vs 爬升 71→132ms），冷 turn1 多付 ~200ms（M3 目标）。含 vllm-bench 命令与 `max_completion_tokens` 坑。 |
 
 ## conventions

diff --git a/docs/models/qwen35/roadmap.md b/docs/models/qwen35/roadmap.md
@@ -47,7 +47,7 @@ out:
 | Fault isolation | Open risk: batch-level execution errors can still fail multiple active requests | #654 |
 | Prefix reuse | Open: bounded joint KV/recurrent/conv snapshot design and implementation | #257 |
 | DFlash | In flight and opt-in: correctness-first work must stay default-off until gates pass | #434, PR #626, #654 |
-| Tensor parallel | Phase 1 complete: eager dense TP2 worker/scheduler execution; Phase 2 still needs mixed-step execution and sharded linear-attention/GDR state. | `docs/models/qwen35/tp-implementation.md`, #446 |
+| Tensor parallel | Done through Phase 2b: eager dense TP2, mixed-step unified execution, and sharded linear/GDR state, verified on 2× RTX 4090 for 9B and 27B; TP CUDA Graph + perf gates remain | `docs/models/qwen35/tp-implementation.md`, #446 |
 
 ## Active Contract
 

diff --git a/docs/models/qwen35/tp-design.md b/docs/models/qwen35/tp-design.md
@@ -1,6 +1,6 @@
 # Qwen3.5 Tensor Parallelism Design
 
-> **TL;DR:** Qwen3.5 TP Phase 2 is two separately delivered correctness milestones: P2a adds eager `RunUnifiedStep` with a shared ordered `RequestId` plan while retaining Phase 1 replicated GDR; P2b shards the head-indexed linear-attention/GDR surface and adds only the hidden all-reduce after local `out_proj`.
+> **TL;DR:** Qwen3.5 tensor parallelism should reuse Qwen3's controller/worker TP runtime and stay degree-parametric. Phases 1, 2a, and 2b are implemented (see `tp-implementation.md` for the landing record, including the rebase onto #870): eager dense TP, TP mixed-step unified execution, and sharded linear-attention/GDR state. Remaining design work: TP CUDA Graph capture.
 >
 > **Last touched:** 2026-08
 
@@ -269,6 +269,30 @@ Validation scope:
 - recurrent-state cleanup on finish/drop/cancellation
 - no stale local recurrent state after a new `RequestId` is admitted
 
+## P2c: CUDA Graph under TP
+
+Status: landed (2026-08-20) on `feat/qwen35-tp2-rebased`, gated on
+`local_decode_group_is_compiled` — 4B/9B TP2 capture and replay decode graphs;
+27B TP2 (group 6) stays on the batched eager path byte-for-byte until group-6
+batch-decode kernels are compiled. Execution record: `tp-implementation.md`
+section "P2c — CUDA Graph under TP".
+
+**Gate**: graph mode active iff `enable_cuda_graph && config.local_decode_group_is_compiled(tp)`. 27B TP2 is group-6 (`SUPPORTED_GQA_GROUP_SIZES = [1,2,3,4,8]`, group ratio is TP-invariant), so 27B TP2 keeps the batched eager path byte-for-byte until group-6 batch-decode kernels are compiled; 4B/9B TP2 capture graphs. Startup logs once when graph was requested but the group gate keeps decode eager.
+
+**State model**: scheduler owns slot semantics (TP1 mirror); workers execute slot copies on command, never infer slots worker-side.
+
+- KV paged state unchanged (pool stable; page tables are per-step H2D via `sync_paged_meta`).
+- Per rank: `BatchDecodeGraphState`-equivalent at `bucket_for(effective_max_batch)` slots — fixed-address `slot_states: Vec<RecurrentState>` + one persistent `LinearStatePointerTables` built once over slots (contents stable → replay-safe).
+- Admission: decode command rows carry explicit `slot_idx` (`slot_for_new_request`); first decode row D2D-copies prefill `RecurrentState` into the slot (`copy_state_to_slot`), drops the per-request allocation.
+- Retirement: `DropRequest` gains `compaction: Option<(RequestId, from, to)>`; worker D2D-moves slot state (`move_slot_within`), asserts occupancy, poisons on mismatch.
+- Decode rows arrive dense slot order `0..bs`; padding rows clobber free slots (benign — admission overwrites).
+
+**Capture/replay**: startup pre-capture sweep ported from qwen3 (`executor.rs:1424`): `Warmup` (port `warmup_tp_collective`, one all-reduce per bucket message size — lazy NCCL connect inside capture wedges), `Capture`/`Launch` per bucket `[1,2,4,8,16,32,64]` with synthetic rows, `Finalize` asserts all captured; dedicated 600 s abort watchdog (60 s startup timeout too small). New `TpWorkerCommand::Precapture { phase }` via existing exact-rank dispatch. Serve time: replay-only (`ensure is_captured` + `launch_captured`), never capture mid-serving. Sampling/logprobs stay rank-0 host-side outside the graph. Mixed ticks: prefill eager + decode replay; collective order canonical per plan. `TpWorkerState` declares graph state before `model` so graphs drop before the NCCL comm (teardown hang precedent qwen3 `executor.rs:3076`).
+
+**Memory** (27B TP2/rank): weights ~17.5 GB + KV pool ~5.9 GiB + slot state reserve ~6.1 GiB + buffers/graphs ~0.3 + scratch/NCCL ~2.5 ≈ 32 GiB → fits 48 GB. 9B TP2 slot state ~1.6 GiB. Loader already reserves `2 × max_batch × bytes_per_request` before sizing KV.
+
+**Validation ladder**: CPU lib suite → TP2 graph HF gate (9B: sequential + bucket-straddling + post-compaction replay vs eager stats) → e2e scheduler graph variant → serving_tp2 graph smoke → 27B TP2 regression unchanged (group-6 stays eager) → per-bucket eager-vs-graph decode benchmark recorded in `bench_snapshots/`.
+
 ## References
 
 - `docs/models/qwen3/tp-design.md`