feat(qwen35): tp2 phase 2a

[Mrtroll486]

Description

Part of issue #446 and #694. Follows pr #627 and #450.

This PR completes Qwen3.5 Tensor Parallel Phase 2A (P2A): eager TP2 mixed prefill/decode execution with a strict cross-rank lifecycle and failure contract.

P2A builds on the existing Phase 1 dense TP implementation. It keeps linear-attention/GDR weights and request state replicated; sharding that surface remains P2B.

What Changed

• Added a canonical, ordered RunUnifiedStep command for eager TP mixed prefill and decode.
• Reused the shared scheduler planner instead of the TP-only eager planner branch.
• Added an all-rank start gate for state-mutating commands:
  • workers execute only after every rank receives the command;
  • partial dispatch cancels the delivered prefix and poisons the replica.
• Hardened worker response validation:
  • requires exactly one response from every rank;
  • validates rank range, uniqueness, reply variant, and the rank-0 result contract.
• Added lifecycle-aware DropAck { existed } handling:
  • MustBeAbsent requires all ranks to report false;
  • MustExist requires all ranks to report true;
  • mixed or uniformly unexpected results are replica-fatal.
• Changed cancellation ordering to:
  • drain -> prune -> publish load -> admission -> plan
• Added fail-closed scheduler recovery:
  • closes and drains submissions after fatal TP failure;
  • sends one terminal error to every unresolved request owner;
  • publishes an idle terminal load snapshot;
  • begins whole-executor teardown without retrying per-request cleanup.
• Added a completion commit boundary:
  • TP EOS/length and immediate-prefill success events are withheld until all-rank cleanup succeeds.
• Replaced positional TP result matching with strict RequestId alignment:
  • shuffled valid artifacts are accepted;
  • unknown, duplicate, non-final, or missing artifacts poison the replica.
• Added pre-load validation for duplicate CUDA ordinals and ordinals outside the fixed 16-device Triton AOT handle table.
• Added lifecycle snapshots and fault-injection coverage for rank-local state, partial dispatch, state divergence, and worker disconnects.

Failure Semantics

Controller-provable structural errors are rejected before dispatch and leave the executor healthy.

After execution is released, worker-local state mismatches, CUDA/NCCL failures, malformed response sets, artifact violations, and lifecycle failures poison the complete TP replica. The scheduler then fails every unresolved request and exits.

Collective or teardown timeout remains process-level fail-stop.

Scope and Non-Goals

This PR does not:

• shard linear-attention/GDR weights, conv state, recurrent state, or scratch buffers;
• add a post-GDR hidden all-reduce;
• enable TP CUDA Graph capture or replay;
• add vocabulary-parallel embedding or lm_head;
• add TP-aware prefix caching;
• claim a performance improvement.

Those state-sharding changes belong to P2B.

Validation

Validated in release mode on 2x RTX 3090 (SM86) with real Qwen3.5-4B weights.

• P2A/TP lifecycle ignored library gates:
  • 14 passed, 0 failed
  • includes 13 TP2 gates and the TP1 cancellation regression
  • covers mixed unified execution, drop/refill, clean re-admission, lifecycle divergence, partial dispatch, and worker disconnect
  • memory-derived TP2 max_batch=64
• TP2 scheduler E2E:
  • 1 passed, 0 failed
  • covers context rejection, greedy/logprobs, sequential and repeated requests, concurrent mixed sampling, consumer drop, and post-drop health
• TP2 HF short/long golden gates:
  • 2 passed, 0 failed
  • short sequential: mean 0.0260, p99 0.1081
  • short batched: mean 0.0267, p99 0.1167
  • long 4097/8192-token replay: mean 0.0228, p99 0.0689
• TP2 OpenAI-compatible HTTP serving smoke:
  • 1 passed, 0 failed
  • covers streaming, non-streaming, concurrent completions, finite logprobs, and TP + CUDA Graph rejection
• Release all-target clippy with -D warnings: passed
• cargo fmt --all -- --check: passed
• git diff --check: passed

Type of Change

• Bug fix (non-breaking change which fixes an issue)
• New feature (non-breaking change which adds functionality)
• Breaking change (fix or feature that would cause existing functionality to not work as expected)
• Documentation update

Checklist

• My code follows the style guidelines of this project (see docs/conventions/coding-style.md).
• I have performed a self-review of my own code.
• I have formatted my commits according to Commitizen conventions.
• I have run the local test suite and all executed tests pass (see CLAUDE.md).

[Mrtroll486]

@codex review

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 10116dc125

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

[xiaguan]

Thanks for putting the Phase 2A path together. The protocol direction looks reasonable, but the PR still needs a smaller, production-verifiable test surface before merge:

1. Please remove low-signal tests that only restate planner/boolean/minimum/variant logic. Keep the tests that materially protect the distributed invariants: partial dispatch, rank divergence, mixed prefill/decode, completion cleanup, drop/refill/readmission, and HF/HTTP lifecycle behavior.
2. CI must compile and run Clippy for pegainfer-qwen35 with the qwen35 feature. The current CPU jobs do not compile this feature, while the meaningful TP2/HF/scheduler/HTTP tests are ignored, so the changed path is not currently a merge gate.
3. Please add and publish a reproducible real TP=2 serving run using vllm bench serve with a multi-turn conversational workload, including the exact command/configuration, request count/concurrency, prompt/output lengths, and pass criteria. The run should exercise mixed request lengths and verify that requests can be cleaned up and admitted again across turns. If the pinned vllm bench serve only flattens a conversation dataset to its first turn, use a small sequential chat driver for the turn dependency instead of reporting a single-turn load test as multi-turn.
4. echo is not part of the current serving contract: both bridges submit echo: false. Reject echo=true before TP dispatch and remove or seal the half-supported PromptTokens path; please state this limitation explicitly in the PR scope.
5. Remove internal absolute model-path fallbacks from ignored tests. Tests should use the repository test helper/model configuration and either fail or skip explicitly when the required public fixture is unavailable.

The important reduction here is not fewer assertions by itself; it is replacing many local self-checks with a small number of real TP2 gates that exercise the same path users will run.

[Mrtroll486]

Thanks for the review. All five follow-ups and the Phase 2A merge gates are now resolved:
Reduced the test surface to the real TP2 distributed-invariant and HF/scheduler/HTTP lifecycle gates (7e0f5fd).
Added Qwen3.5 feature compile and Clippy CI; both hosted SM80/Triton AOT jobs pass (8829189).
Published a reproducible TP2 multi-turn run: 12/12 conversations and 44/44 turns, followed by a no-restart 4/4 conversation readmission probe (4578971).
Rejected echo=true before backend admission or TP dispatch and removed the Qwen3.5 PromptTokens path (0b28916).
Removed private model-path fallbacks and required an explicit validated public fixture (0039a41).
Pinned the benchmark fixture to the official Qwen3.5-4B revision and complete weight hashes (5f8c68f).
The local release, real TP2, HF, scheduler, HTTP, multi-turn, formatting, Clippy, and hosted CI gates are all green.