Runtime CP<->DP mode switching
Summary
Would like to add support for flipping worker layout between CP_PREFILL (cp=N, dp=1) and DP_DECODE (cp=1, dp=N) on a live xLLM instance in ~500 ms, without a service restart. This gives one deployment the ability to adapt across the day rather than being statically sized for either lane.
Motivation
xLLM supports two parallel layouts for a fixed world size:
- CP_PREFILL: context parallelism, minimizes TTFT for long-context requests by splitting KV compute across workers.
- DP_DECODE: data parallelism,
N independent replicas, higher concurrency for short-request traffic.
Traffic patterns rarely stay in one regime forever. Long-context batch ingest at night, chat traffic during the day. Static provisioning per instance forces overprovisioning one lane or accepting degraded metrics on the other.
An autoscaler that can switch layout per instance — not just replicas — closes this gap.
Proposed design
- Startup dual-graph mode: with
--enable_runtime_cp_dp_switch=true, the worker initializes BOTH ATB graphs up front. A flip is a graph handle swap, not a rebuild.
- Flip flow: scheduler drains in-flight requests → pause workers → swap the active
ParallelArgs (cp/dp) and block-manager pool atomically → resume.
- Worker-side call sites read the live
parallel_args_.cp_size() (not options_.cp_size()), so already-scheduled requests observe the new layout after resume.
- RPC surface:
mode_switch.proto + ModeSwitchService exposed via the xLLM server.
- Auto-flip: an
AutoFlipController on the scheduler side that watches traffic stats and issues switch RPCs, so an instance can flip layout autonomously.
- Heal path: bounded heal window after a switch to drain long-prefill tails that straddle the flip.
- Observability: 7 bvar metrics on
/vars (flip totals, target-CP / target-DP totals, latency, heal rounds, pending-gate rejects).
Non-goals
- Not changing static-mode behavior when
--enable_runtime_cp_dp_switch=false. The feature is fully gated.
- Not changing model correctness. lm_eval CP == DP is a hard verification bar.
Scope
Two PRs planned:
- PR1: core runtime feature (dual-graph init, mode switch RPC, atomic pool swap). ~2700 lines, 59 files.
- PR2: AutoFlipController + heal path + bvar metrics on top. ~1100 lines, 16 files.
Plus two prerequisite standalone PRs already isolated for review clarity.
Verification
- Unit:
dual_parallel_args_test, atomic_pool_handle_test, lopsided_defer_gate_test, auto_flip_decide_target_test.
- E2E on 8*A3 (DeepSeek V32, 61 layers):
verify_switch regression 18/18 flips stable, ~500 ms per flip; lm_eval CP == DP within tolerance.
- Soak: 10 min @ 32 procs periodic auto-flips, HBM watermark stable.
Not yet covered
- Cross-machine disagg PD flip (single-node validated).
- Long-run (>1 h) stability.
- Non-DeepSeek-V32 layer variants (layer wiring is architecture-specific in this iteration).
Design doc will land with PR1 under docs/en/features/runtime_cp_dp_switch.md.
Runtime CP<->DP mode switching
Summary
Would like to add support for flipping worker layout between CP_PREFILL (
cp=N, dp=1) and DP_DECODE (cp=1, dp=N) on a live xLLM instance in ~500 ms, without a service restart. This gives one deployment the ability to adapt across the day rather than being statically sized for either lane.Motivation
xLLM supports two parallel layouts for a fixed world size:
Nindependent replicas, higher concurrency for short-request traffic.Traffic patterns rarely stay in one regime forever. Long-context batch ingest at night, chat traffic during the day. Static provisioning per instance forces overprovisioning one lane or accepting degraded metrics on the other.
An autoscaler that can switch layout per instance — not just replicas — closes this gap.
Proposed design
--enable_runtime_cp_dp_switch=true, the worker initializes BOTH ATB graphs up front. A flip is a graph handle swap, not a rebuild.ParallelArgs(cp/dp) and block-manager pool atomically → resume.parallel_args_.cp_size()(notoptions_.cp_size()), so already-scheduled requests observe the new layout after resume.mode_switch.proto+ModeSwitchServiceexposed via the xLLM server.AutoFlipControlleron the scheduler side that watches traffic stats and issues switch RPCs, so an instance can flip layout autonomously./vars(flip totals, target-CP / target-DP totals, latency, heal rounds, pending-gate rejects).Non-goals
--enable_runtime_cp_dp_switch=false. The feature is fully gated.Scope
Two PRs planned:
Plus two prerequisite standalone PRs already isolated for review clarity.
Verification
dual_parallel_args_test,atomic_pool_handle_test,lopsided_defer_gate_test,auto_flip_decide_target_test.verify_switchregression 18/18 flips stable, ~500 ms per flip; lm_eval CP == DP within tolerance.Not yet covered
Design doc will land with PR1 under
docs/en/features/runtime_cp_dp_switch.md.