feat(qwen35): cuda graph decode under TP (split 3/4 of #946) - #1005

Move the per-rank worker runtime out of the tp_executor.rs God module:
TpWorker + spawn/drop lifecycle, TpStartupGate, the NCCL startup watchdog,
TpWorkerState/TpWorkerPrepared and its command loop (run/respond/
execute_*/precapture/drop), the slot bookkeeping helpers, the shared
decode-row view/sampling helpers, recurrent-capacity math, and the worker
CublasThreadGuard binding. The entry keeps command/reply types, the
executor orchestration (including the precapture sweep), plan validators,
and response validation, reaching worker items via the same `use super::*`
style as scheduler/{backend,tp}. Rebased over pegainfer-project#1004/pegainfer-project#1005/pegainfer-project#1033/pegainfer-project#1038.
Behavior unchanged.

Retire the watchdog disarm wiring test: the happy path is exercised by
every TP e2e startup and the timeout path process-aborts by design, so it
has no assertable in-process failure mode.

…es.rs

Move the reply envelope (TpWorkerReply/TpWorkerResponse), bounded response
collection, per-command response validation, and the step-timeout/abort
helpers out of tp_executor.rs; the three response-contract tests move with
them. Behavior unchanged, with two behavior-preserving consolidations in
the moved code: validate_exact_rank_responses folds its push loop into an
iterator pipeline, and the three rank-0 payload validators (prefill /
decode / unified) collapse into one validate_primary_responses helper over
payload extractors, keeping the exact prior error messages. Rebased over
pegainfer-project#1004/pegainfer-project#1005/pegainfer-project#1033/pegainfer-project#1038 (the precapture sweep keeps using
validate_ack_responses, which stays production-visible).

Move the per-rank worker runtime out of the tp_executor.rs God module:
TpWorker + spawn/drop lifecycle, TpStartupGate, the NCCL startup watchdog,
TpWorkerState/TpWorkerPrepared and its command loop (run/respond/
execute_*/precapture/drop), the slot bookkeeping helpers, the shared
decode-row view/sampling helpers, recurrent-capacity math, and the worker
CublasThreadGuard binding. The entry keeps command/reply types, the
executor orchestration (including the precapture sweep), plan validators,
and response validation, reaching worker items via the same `use super::*`
style as scheduler/{backend,tp}. Rebased over pegainfer-project#1004/pegainfer-project#1005/pegainfer-project#1033/pegainfer-project#1038.
Behavior unchanged.

Retire the watchdog disarm wiring test: the happy path is exercised by
every TP e2e startup and the timeout path process-aborts by design, so it
has no assertable in-process failure mode.

Signed-off-by: CAICAIIs <3360776475@qq.com>

…es.rs

Move the reply envelope (TpWorkerReply/TpWorkerResponse), bounded response
collection, per-command response validation, and the step-timeout/abort
helpers out of tp_executor.rs; the three response-contract tests move with
them. Behavior unchanged, with two behavior-preserving consolidations in
the moved code: validate_exact_rank_responses folds its push loop into an
iterator pipeline, and the three rank-0 payload validators (prefill /
decode / unified) collapse into one validate_primary_responses helper over
payload extractors, keeping the exact prior error messages. Rebased over
pegainfer-project#1004/pegainfer-project#1005/pegainfer-project#1033/pegainfer-project#1038 (the precapture sweep keeps using
validate_ack_responses, which stays production-visible).

Signed-off-by: CAICAIIs <3360776475@qq.com>

Move the per-rank worker runtime out of the tp_executor.rs God module:
TpWorker + spawn/drop lifecycle, TpStartupGate, the NCCL startup watchdog,
TpWorkerState/TpWorkerPrepared and its command loop (run/respond/
execute_*/precapture/drop), the slot bookkeeping helpers, the shared
decode-row view/sampling helpers, recurrent-capacity math, and the worker
CublasThreadGuard binding. The entry keeps command/reply types, the
executor orchestration (including the precapture sweep), plan validators,
and response validation, reaching worker items via the same `use super::*`
style as scheduler/{backend,tp}. Rebased over pegainfer-project#1004/pegainfer-project#1005/pegainfer-project#1033/pegainfer-project#1038.
Behavior unchanged; the watchdog disarm test moves into worker.rs with
its implementation.

Signed-off-by: CAICAIIs <3360776475@qq.com>

…es.rs

Move the reply envelope (TpWorkerReply/TpWorkerResponse), bounded response
collection, per-command response validation, and the step-timeout/abort
helpers out of tp_executor.rs; the three response-contract tests move with
them. Behavior unchanged, with two behavior-preserving consolidations in
the moved code: validate_exact_rank_responses folds its push loop into an
iterator pipeline, and the three rank-0 payload validators (prefill /
decode / unified) collapse into one validate_primary_responses helper over
payload extractors, keeping the exact prior error messages. Rebased over
pegainfer-project#1004/pegainfer-project#1005/pegainfer-project#1033/pegainfer-project#1038 (the precapture sweep keeps using
validate_ack_responses, which stays production-visible).

Signed-off-by: CAICAIIs <3360776475@qq.com>

…es.rs

Move the reply envelope (TpWorkerReply/TpWorkerResponse), bounded response
collection, per-command response validation, and the step-timeout/abort
helpers out of tp_executor.rs; the three response-contract tests move with
them. Behavior unchanged, with two behavior-preserving consolidations in
the moved code: validate_exact_rank_responses folds its push loop into an
iterator pipeline, and the three rank-0 payload validators (prefill /
decode / unified) collapse into one validate_primary_responses helper over
payload extractors, keeping the exact prior error messages. Rebased over
pegainfer-project#1004/pegainfer-project#1005/pegainfer-project#1033/pegainfer-project#1038 (the precapture sweep keeps using
validate_ack_responses, which stays production-visible).

Signed-off-by: CAICAIIs <3360776475@qq.com>

…es.rs

Move the reply envelope (TpWorkerReply/TpWorkerResponse), bounded response
collection, per-command response validation, and the step-timeout/abort
helpers out of tp_executor.rs; the three response-contract tests move with
them. Behavior unchanged, with two behavior-preserving consolidations in
the moved code: validate_exact_rank_responses folds its push loop into an
iterator pipeline, and the three rank-0 payload validators (prefill /
decode / unified) collapse into one validate_primary_responses helper over
payload extractors, keeping the exact prior error messages. Rebased over
pegainfer-project#1004/pegainfer-project#1005/pegainfer-project#1033/pegainfer-project#1038 (the precapture sweep keeps using
validate_ack_responses, which stays production-visible).

Signed-off-by: CAICAIIs <3360776475@qq.com>

…it 4/4 of #946) (#1006)

* feat(qwen35): cuda graph decode under TP

P2c from docs/models/qwen35/tp-design.md. TP decode captures/replays
per-bucket CUDA Graphs when --cuda-graph is set AND the TP-local decode
GQA group has a compiled kernel (4B/9B TP2); 27B TP2 (group 6) keeps the
batched eager path byte-for-byte under the gate.

- Gate: drop the fail-closed TP+graph rejections in config.rs/lib.rs/
  tp_executor.rs; log once when graph was requested but the group gate
  keeps decode eager.
- State: scheduler owns dense decode slots (slot_idx on TP decode rows,
  slot_for_new_request at promote, compaction_after_retire on retire);
  workers hold a fixed-address BatchDecodeGraphState plus slot_map, D2D
  copy prefill state into the slot on the first decode row, and apply
  DropRequest compactions via move_slot_within with occupancy assertions
  (poison on mismatch).
- Capture/replay: startup pre-capture sweep ported from qwen3 (Warmup
  via new warmup_tp_collective, Capture/Launch per bucket, Finalize)
  behind a 600 s abort watchdog; batch_decode_graph gains DecodeGraphUse
  (Serve/CaptureOnly/Replay); TP serving is replay-only; graphs drop
  before the NCCL comm.
- Convenience executor API (model-local tests) keeps a slot tracker
  mirroring Qwen35Executor; scheduler flows pass explicit slots.

Validation (2x RTX 4090, venv NCCL): lib 105/105; 9B TP2 HF gates eager
+ graph (sequential replay, bucket-straddling, post-compaction) pass
within existing tolerances; 9B TP2 scheduler e2e eager+graph pass;
serving_tp2 now launches with graph on; 27B TP2 HF+e2e pass unchanged
(group-6 gate keeps eager, graph variant self-skips). Serving benchmark
9B TP2, 16 concurrent x 256 out: 767.15 tok/s graph vs 705.86 eager
(+8.7% steady output, TPOT 20.04 vs 21.99 ms).

Signed-off-by: Ziyang Zhang <hafuhafu@qq.com>

* fix(qwen35): size TP graph slot reserve by effective batch bucket

The graph-slot pre-reserve used bucket_for(requested_max_batch), but the
graph state is later allocated as bucket_for(effective_max_batch). On a
tight-memory rank the oversized reserve could starve the effective
recurrent capacity to zero and refuse startup.

Iterate the reserve bucket downward against the fitted capacity until it
stabilises (the bucket only shrinks, so it converges), and clamp the
fitted batch to the reserved bucket so the later bucket_for(effective)
allocation never exceeds the reserve.

Absorbs the codex review comment on Ma1oneZhang/pegainfer PR #946
(tp_executor.rs graph_slot_reserve).

Signed-off-by: Ziyang Zhang <hafuhafu@qq.com>

* refactor(qwen35): point the TP graph gate at the config-level decode-group predicate

The rank-local GQA group equals the global group under validated head
sharding, so the P2B review removed LocalGeometry::local_decode_group_is_compiled.
Route the P2c graph gate through Config35::decode_group_is_compiled instead
of re-adding the local duplicate; the gate decision was already identical on
every rank.

Evidence (2x RTX 4090, sm_89): cargo check/clippy --release --all-targets
-D warnings clean; qwen35 lib tests 107 passed / 0 failed; cargo fmt clean.

Signed-off-by: Ziyang Zhang <hafuhafu@qq.com>

* docs(qwen35): retarget the TP record at the upstream module splits

The rebase onto main lands this branch's TP deltas in the layout
#967/#968/#969 introduced, so the record names where the code actually
lives: linear-attention sharding in weights/layers.rs on WeightSource,
and the TP slot bookkeeping across scheduler/{mod,backend,tp}.rs.
Paths only — no invariant changed.

Signed-off-by: Ziyang Zhang <hafuhafu@qq.com>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* fix(qwen35): TP pre-capture sweep holds one KV page, not one per bucket row

precapture_bucket allocated a real KvState for every row of the bucket
being captured, so a rank whose usable KV pages were fewer than
bucket_for(max_batch) failed at startup even though serving would have
padded those rows. The sweep now passes one real row and the bucket
explicitly through batch_decode_graph_padded; every other row is padding
on the pool's reserved padding page, exactly as when serving, so the
sweep's page demand is a constant 1 regardless of pool size or bucket.
batch_decode_graph is unchanged for its callers (it delegates with
bucket_for(bs)); the padded entry point checks the bucket covers bs.

Evidence: 9B TP2 HF gates on 2x RTX 4090 (eager, graph incl. 5->8 and
3->4 padded buckets and post-compaction replay, long) 3/3 pass, 39 s.
Codex review follow-up on #1005.

Signed-off-by: Ziyang Zhang <hafuhafu@qq.com>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* docs(qwen35): P2c record without a Status field; label the pre-rebase A/B

tp-design.md: the P2c section's "Status:" line becomes dated prose (the
doc convention forbids status enums), the stale branch name goes, and the
TL;DR gains the landed P2c gate so the header matches the body.
tp-implementation.md: the 9B TP2 767-vs-706 tok/s table is marked as the
pre-rebase #946 measurement it is; no throughput claim is made for this
stack until the same-context A/B is rerun on it (#1001).
Codex review follow-up on #1005.

Signed-off-by: Ziyang Zhang <hafuhafu@qq.com>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* docs(qwen35): record the TP rebase onto #870

Signed-off-by: Ziyang Zhang <hafuhafu@qq.com>

* docs(qwen35): TP implementation record + 27B TP2 knowledge benchmark eval

- tp-implementation.md: Phase 2a/2b + Step 3 batched-decode landing record
- benchmarks/qwen35-27b-tp2-knowledge-eval.md: MMLU-Redux 94.09 vs 93.2,
  C-Eval 88.11 vs 90.5 (full runs, truncation rerun-merged); in band
- scripts/eval_mc.py + eval_rerun_truncated.py: benchmark runner
  (OpenCompass/lm-eval recipe replicas over the chat completions API)
- developer-onboarding: NCCL libnccl.so dlopen note

Signed-off-by: Ziyang Zhang <hafuhafu@qq.com>

* fix(bench): mmlu-redux extracts answers by markers, not first capital

The MMLU-Redux extractor searched the whole completion for the first
[ABCD], so any answer prefixed with "Answer: B" scored as the "A" in
"Answer", silently marking correct B/C/D responses wrong (codex review
on #946). Reuse the SuperGPQA marker-preferring patterns restricted to
A-D, and drop the now-dead MMLUREDUX_RE. Redo the MMLU-Redux numbers
before citing them.

Signed-off-by: Ziyang Zhang <hafuhafu@qq.com>

* docs(bench): flag the pre-fix MMLU-Redux figure as rerun-pending

The 94.09 snapshot predates the answer-extraction fix; mark it as not
citable until rerun with the marker-preferring extractor.

Signed-off-by: Ziyang Zhang <hafuhafu@qq.com>

* docs(qwen35): derive value-head TP divisibility from the key-head guard

The second LocalGeometry guard was dropped in review: Config35 already
validates linear_num_value_heads % linear_num_key_heads == 0, so the
key-head % tp guard implies value-head divisibility. Align the P2/P2b
requirement wording with the single source of truth.

Signed-off-by: Ziyang Zhang <hafuhafu@qq.com>

* fix(bench): rerun script uses the marker-aware MMLU-Redux extractor; SuperGPQA records keep options

eval_rerun_truncated.py imported MMLUREDUX_RE, which eval_mc.py dropped
when MMLU-Redux moved to sg_extract_labels, so every rerun invocation
died with ImportError before parsing arguments. It now calls the same
marker-aware extractor (letters ABCD). Its SuperGPQA content fallback
also never fired because the sample records eval_mc.py writes carried no
'options'; the records now keep the option texts.
Codex review follow-up on #1006.

Signed-off-by: Ziyang Zhang <hafuhafu@qq.com>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* docs(bench): flag the MMLU-Redux 94.09 index row as rerun-pending

The routing row cited 94.09 as an in-band result while the linked report
says it came from the pre-fix first-capital extractor and must not be
cited before a rerun. Codex review follow-up on #1006.

Signed-off-by: Ziyang Zhang <hafuhafu@qq.com>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* fix(bench): address codex P2 review on #1006

- eval_mc: sg_extract_content accepts 'Answer: <option text>' (colon after
  the Answer marker); --sample >= dataset size with default --limit now
  keeps the full dataset instead of silently evaluating zero rows
- knowledge-eval doc TL;DR + body: mark the MMLU-Redux 94.09 figure as
  extracted with the pre-fix first-capital extractor, rerun-pending —
  no longer cited as in-band parity evidence
- roadmap: TP row advanced through P2c (4B/9B TP2 graph landed; 27B
  group-6 remains eager), freshness stamp 2026-07 -> 2026-09

Signed-off-by: Ziyang Zhang <hafuhafu@qq.com>

* fix(bench): eval scripts — no gen-time stops, fail-fast exit, per-sample persistence, config-checked reruns

- eval_mc: drop the 'Question:' generation-time stop for mmlu_pro — the
  stack matches stop text against the raw stream before reasoning/final
  separation, so a thinking-model repetition ends generation early
  (eval_gsm8k_thinking.py precedent)
- eval_mc: persist each sample to <bench>_samples.partial.jsonl as it
  arrives instead of keeping everything in memory until the end
- eval_mc: a run with requests still failing after retries is marked
  'incomplete' in the summary and exits 2 — a dead service can no longer
  produce a normal-looking low-accuracy report
- eval_rerun_truncated: inherit model and temperature from the initial
  run's summary (recorded as rerun_model/rerun_temperature); an explicit
  --model that disagrees with the initial run is refused instead of
  silently merging configurations

Signed-off-by: Ziyang Zhang <hafuhafu@qq.com>

* fix(bench): rerun script persists per-row + drop weak bogus-path TP gate smokes

- eval_rerun_truncated: persist every rerun row to
  <bench>_rerun.partial.jsonl as it completes; a request that exhausts
  retries now lands back in the merged output as an explicitly
  '__ERROR__'-marked empty row instead of raising through gather and
  discarding the batch's completed (expensive, large-budget) responses;
  summary records rerun_errors and exits 2 when any remain
- qwen35: drop the three 'bogus path does not say eager execution only'
  startup tests (lib.rs, scheduler/tests.rs, tp_executor.rs): the
  assertion passes for unrelated failures such as early CUDA init errors
  and does not pin the P2c gate; the real TP2 serving test covers the
  entry point with enable_cuda_graph=true; unused imports removed with
  them, tp-implementation record wording restored to the slots-test set

Signed-off-by: Ziyang Zhang <hafuhafu@qq.com>

* fix(bench): codex round-3 P2s — exact strata sampling, rerun summary failure-state recompute

- eval_mc --sample: allocate exactly the requested total across strata
  (largest remainder, capacity-respecting) instead of round()+max(1) per
  group, which ran 57 rows for --sample 10 on 57-subject MMLU-Redux and
  overshot larger targets; the recorded n now matches the requests made
- eval_rerun_truncated: after a fully-failed row is fixed on rerun,
  clear 'incomplete' and recompute api_errors from the merged rows;
  initial-run counts live on under initial_api_errors
- tp-implementation Follow-Ups: the pre-fix MMLU-Redux figure is no
  longer cited as in-band parity evidence; C-Eval is the current evidence
- roadmap Later: replace the already-landed 'TP for a concrete 9B/27B
  target' entry with the actual remaining TP work (group-6 batch-decode
  kernels, perf gates) pointing at tp-implementation.md

Signed-off-by: Ziyang Zhang <hafuhafu@qq.com>

* fix(bench): codex round-4 P2s — longest-first option match, rerun resumes from merged state

- eval_mc sg_extract_content: order alternatives longest-first so a
  shorter option that is a prefix of a longer one ('New York' vs
  'New York City') can no longer match prematurely and flip the letter
- eval_rerun_truncated: a follow-up invocation resumes from
  <bench>_samples_merged.json when present, so only rows still empty are
  regenerated and a previously successful large-budget completion can
  never be replaced by a new failure; a successful rerun now also clears
  the carried-over rerun_failed marker
- knowledge-eval doc: the results-table Redux verdict no longer claims
  in-band parity ('无带内判定:修复前抽取器打分,复跑待定勿引用'); the
  C-Eval gap-to-truncation attribution is now explicitly a hypothesis —
  the non-truncated subset is self-selected and truncation correlates
  with difficulty, so an uncapped same-context comparison is still owed

Signed-off-by: Ziyang Zhang <hafuhafu@qq.com>

* fix(bench): rerun resume hardening — stale-merged guard, initial-error history, partial recovery

- rerun resume (P1): a fresh eval_mc run in the same directory now
  unlinks <bench>_samples_merged.json and the rerun partial checkpoint,
  so an old merged state can never be attributed to a new run; and the
  rerun script itself refuses to resume from a merged file whose length
  or per-row prompts do not match the current base samples
- retry chains preserve history: initial_api_errors is kept from the
  first rerun record instead of being overwritten by the remaining-error
  count of an intermediate attempt (5 -> 2 -> 0 no longer records '2
  initial')
- killed reruns recover: the partial JSONL is read (skipping a torn tail
  line) and appended to rather than truncated, so completions from an
  interrupted attempt are reused and only still-empty rows regenerate

Signed-off-by: Ziyang Zhang <hafuhafu@qq.com>

* docs(bench): TP+graph environment note reflects the P2c gate

The environment line still claimed TP + CUDA Graph is fail-closed. Since
P2c the graph/eager decision is a compiled-decode-GQA-group gate: 4B/9B
TP2 capture graphs, and a 27B group-6 request silently keeps the eager
fallback instead of failing startup.

Signed-off-by: Ziyang Zhang <hafuhafu@qq.com>

* fix(bench): eval checkpoint is resumable — recover completed rows before append

The per-sample partial JSONL was opened with 'w' on every invocation, so
a killed long evaluation lost its flushed completions on the next run:
the checkpoint was write-only. Reuse the rerun script's approach: read
back recovered rows (skip torn tail lines; require a prompt match for the
current invocation; skip __ERROR__ rows), append new records instead of
truncating, and fold recovered truncation/correctness into the summary
counts.

Signed-off-by: Ziyang Zhang <hafuhafu@qq.com>

* docs(qwen35): Phase 2 progress paragraph reflects the landed P2c state

The progress statement still sent readers toward TP CUDA Graph as the
remaining work although the document's own P2c section records it landed
for compiled decode GQA groups. Point at the actual remainder: group-6
batch-decode kernels for 27B TP2 graphs, and the TP perf gates.

Signed-off-by: Ziyang Zhang <hafuhafu@qq.com>

* docs(bench): eval_mc module docstring tracks the marker-aware Redux extractor

Signed-off-by: Ziyang Zhang <hafuhafu@qq.com>

* fix(bench): eval checkpoint resume is gated on a run-configuration fingerprint

A reused output directory with different --model/--temperature/--max-tokens
produces identical prompts, so the prompt-equality resume predicate accepted
rows generated under the old settings while the summary reported the new
arguments. Persist a fingerprint of the generation settings next to the
checkpoint and only resume when it matches; otherwise discard the checkpoint
and start fresh.

Signed-off-by: Ziyang Zhang <hafuhafu@qq.com>

* fix(bench): keep eval checkpoints resumable without trusting stale verdicts

- eval_mc rescores recovered rows with the current extractor and golds:
  the generated text is reused, but saved pred/gold/correct are not trusted,
  so a fixed extractor or corrected dataset answer lands without regeneration
- Both runners repair the checkpoint before appending: a torn tail from a
  killed write no longer swallows the next appended record, and malformed
  complete lines are reported as corruption instead of silently skipped
- eval_mc agg_toks_per_s now pairs tokens with the interval that produced
  them (this run's completions only) and is omitted on a fully cached resume
- eval_rerun_truncated scores pred/correct before persisting a rerun row, so
  an interrupted run never leaves new text next to stale verdicts

Signed-off-by: Ziyang Zhang <hafuhafu@qq.com>

---------

Signed-off-by: Ziyang Zhang <hafuhafu@qq.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>