Skip to content

[Feature] Add EPD routing with JSON metadata - #266

Draft
gty111 wants to merge 1 commit into
vllm-project:mainfrom
gty111:epd-json-proxy
Draft

[Feature] Add EPD routing with JSON metadata#266
gty111 wants to merge 1 commit into
vllm-project:mainfrom
gty111:epd-json-proxy

Conversation

@gty111

@gty111 gty111 commented Sep 10, 2026

Copy link
Copy Markdown

Purpose

Add E+PD and E/P/D orchestration through --epd-config, integrating encoder routing into vLLM Router instead of requiring a separate Python EPD proxy.

  • Select P/PD before encoding, distribute media across E workers, and forward JSON metadata without PyTorch or tensor deserialization.
  • Keep embedding transfers in EC backends; support Mooncake PUSH control and opaque NIXL/example handles. Reuse the existing P/D KV handoff, without forwarding P's EC handles to D.
  • Preserve sampling fields and SSE responses; cancel abandoned Mooncake transfers.

Native CLI, static workers, DP=1 initially. Requires vLLM JSON metadata support (vllm-project/vllm#56090). See usage and limitations.

Related to #34. No equivalent open PR found: #103 selects modality-specific prefill pools rather than orchestrating separate encoders.

Validation

Native HTTP/SSE smoke passed for E+PD and E/P/D with mock backends, including duplicate images and encoder failures.

Earlier GPU runs: Qwen3.5-35B-A3B, Mooncake EC RDMA, concurrency 96, CUDA graphs through 16384 (no eager), structured choice; 200 warmup requests plus five repetitions of the first 1,200 MUIRBench questions, with caches retained.

4E + 4PD: existing Python EPD proxy vs. vLLM Router

Metric Python proxy (historical) vLLM Router Change
Mean throughput (req/s) 61.40 93.33 +52.0%
Mean request latency (ms) 1514.0 1019.0 -32.7%
Pooled P99 latency (ms) 5007.1 3051.0 -39.1%
Accuracy, all final answers 58.93% 59.13% +0.20 pp

Same model, client settings and 8-GPU topology, with the proxy on E's node. The Python baseline was not rerun; different nodes/times prevent attributing the entire speedup to the router. Router had one recovered HTTP 502; latency and accuracy include that final response.

Separate 4E + 4P + 4D validation with NIXL KV achieved 90.79 req/s, 58.98% accuracy, with no request failures. This uses 12 GPUs and is not an equal-resource comparison with 4E + 4PD.

These are subset evaluations, not full MUIRBench scores. GPU runs predate the final code cleanup and have not been rerun afterward.

AI assistance was used for implementation and testing.

Signed-off-by: Tianyu Guo <guoty@inferact.ai>
@gty111 gty111 changed the title [Feature] Add experimental EPD routing with JSON metadata [Feature] Add EPD routing with JSON metadata Sep 10, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant