[Feature] Add EPD routing with JSON metadata - #266
Draft
gty111 wants to merge 1 commit into
Draft
Conversation
Signed-off-by: Tianyu Guo <guoty@inferact.ai>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Purpose
Add E+PD and E/P/D orchestration through
--epd-config, integrating encoder routing into vLLM Router instead of requiring a separate Python EPD proxy.Native CLI, static workers, DP=1 initially. Requires vLLM JSON metadata support (vllm-project/vllm#56090). See usage and limitations.
Related to #34. No equivalent open PR found: #103 selects modality-specific prefill pools rather than orchestrating separate encoders.
Validation
Native HTTP/SSE smoke passed for E+PD and E/P/D with mock backends, including duplicate images and encoder failures.
Earlier GPU runs: Qwen3.5-35B-A3B, Mooncake EC RDMA, concurrency 96, CUDA graphs through 16384 (no eager), structured choice; 200 warmup requests plus five repetitions of the first 1,200 MUIRBench questions, with caches retained.
4E + 4PD: existing Python EPD proxy vs. vLLM Router
Same model, client settings and 8-GPU topology, with the proxy on E's node. The Python baseline was not rerun; different nodes/times prevent attributing the entire speedup to the router. Router had one recovered HTTP 502; latency and accuracy include that final response.
Separate 4E + 4P + 4D validation with NIXL KV achieved 90.79 req/s, 58.98% accuracy, with no request failures. This uses 12 GPUs and is not an equal-resource comparison with 4E + 4PD.
These are subset evaluations, not full MUIRBench scores. GPU runs predate the final code cleanup and have not been rerun afterward.
AI assistance was used for implementation and testing.