[Feature] Qwen3-14B single-node multi-card serving with DP and TP

[bumble0918]

Summary

Bring up Qwen3-14B single-node multi-card serving inference with DP and TP parallelism, complete functional generalization tests, and collect initial performance baseline numbers.

Motivation / Use Case

The previous Qwen3-14B serving milestone focused on core serving capabilities and single-card validation. The next serving milestone should validate that the same serving stack can scale to a single-node multi-card deployment and exercise both data parallel and tensor parallel execution modes.

This is the practical next step from a single-card runnable baseline toward deployable serving capacity.

Proposed API / Behavior

• Qwen3-14B serving runs on a single machine with multiple cards.
• DP mode is supported and validated.
• TP mode is supported and validated.
• Functional generalization tests cover representative request shapes, batch sizes, prefix-cache behavior and decode/prefill paths.
• Performance numbers are collected for throughput, TTFT, TPOT/P99 latency, prefix-cache hit rate and KV cache utilization.

requests
  -> scheduler / batching / prefix cache
  -> Qwen3-14B multi-card serving
  -> DP and TP execution
  -> correctness + performance report

Related Work

• meta-sprint#4 — Qwen3-14B single-card serving validation
• pypto-serving#37 — v1 parallel serving strategy support
• pypto-serving#38 — radix prefix cache backend for paged KV reuse
• pypto-serving#39 — radix prefix cache backend
• pypto-serving#41 — Qwen3-14B prefix-cache crash report
• pypto-serving#43 — Qwen3-14B generalization validation report

Closure Note For Previous Milestone

meta-sprint#4 appears to have reached the original single-card/core-capability scope if the benchmark baseline artifacts are available. This issue tracks the follow-up multi-card serving scope separately.

Acceptance

• DP serving path runs end-to-end on a single multi-card node.
• TP serving path runs end-to-end on a single multi-card node.
• Functional generalization test report is attached or linked.
• Initial performance report is attached or linked.
• Known crash/regression cases are either fixed or explicitly documented with follow-up issues.