[WIP][Help Wanted] vLLM Router 2026 Roadmap

[fake0fan]

vLLM Router 2026 Roadmap Issues

Planning date: September 8, 2026
Code baseline: vLLM Router main@1d10e71 (identical to upstream main at the time of review)
Objective: define an actionable roadmap for the remainder of 2026 around vLLM-native integration, lightweight high-performance routing, exact KV awareness, and agent-aware scheduling.
Scope: CPU fast path, exact KV routing, agent-aware scheduling, E/P/D, and Router HA.

Overview

#	Roadmap issue	Current foundation	Priority	2026 target	External value
1	Integrate and Optimize the CPU-side Inference Fast Path	◐ Rust tokenizer, streaming utilities, and benchmarks exist, but they do not yet form an end-to-end request fast path	P0	Tokenizer cache, request-scoped reuse, and JSON/streaming hot-path optimization	Demonstrate the performance advantage of a lightweight Rust router
2	Event-driven Exact KV-aware Routing	○ Exact KV awareness is missing; the current cache-aware policy uses an approximate request-history tree	P0	KV Events, an exact index, unified cost, and safe fallback	Flagship capability for the year
3	Extend Session Affinity to Agent/Program-aware Scheduling	◐ Session-ID-based consistent hashing exists	P1	Program metadata, a program-aware policy, and a KV TTL prototype	Differentiate for agentic workloads
4	Extend P/D Routing to E/P/D	◐ P/D, DP-rank routing, and KV connectors exist; the Encode stage does not	P1	Encode pool and three-stage routing MVP	Extend multimodal scheduling
5	Complete Multi-Router HA and State Recovery	◐ Health checks, retries, readiness, and process-level graceful shutdown exist; multi-Router state HA does not	P0 foundation / P1 complete	Drain and recovery foundations; active-active as a stretch goal	Make stateful routing production-ready

Roadmap principles

• Do not turn vLLM Router into a broad, all-in-one gateway; its core remains vLLM inference routing.
• Prioritize CPU optimizations that directly benefit tokenization, routing, and the streaming fast path.
• Make exact KV state trustworthy before optimizing policy gains.
• Agent hints must be optional, and ordinary OpenAI-compatible requests must remain compatible.
• Deliver minimal closed loops for EPD and HA first rather than expanding the initial issues into full platform projects.
• Base every performance claim on reproducible experiments using the same hardware, model, and trace.

---

Issue 1: [Roadmap][P0] Integrate and Optimize the CPU-side Inference Fast Path

Background

As GPU inference becomes faster, CPU work such as tokenization, JSON encoding/decoding, and stream processing can become the bottleneck. SMG moves tokenization and detokenization into a Rust gateway and uses an L0 exact-match tokenizer cache, an L1 prefix-aware tokenizer cache, and a gRPC tokens-in/tokens-out path to reduce Python/GIL and serialization overhead.

vLLM Router already includes a Rust tokenizer, chat-template support, streaming decoding utilities, and benchmark infrastructure. However, the tokenizer module is not yet part of a complete CPU pipeline across the main request path, and there is no L0/L1 token-result cache. The worker data plane is currently HTTP; the gRPC/Tonic dependency in Cargo is primarily used for OpenTelemetry and does not constitute a worker gRPC fast path.

The goal of this issue is to integrate and optimize the existing modules into a measurable CPU fast path. It is not to reimplement the Rust tokenizer or to build a complete tool, MCP, or provider gateway.

Scope

• Establish stable baselines for Router CPU usage, latency, throughput, and memory.
• Integrate the existing Rust tokenizer, chat template, and streaming decoder into the request paths that need them.
• Add a bounded L0 exact-match tokenizer cache.
• Evaluate and implement an L1 prefix token cache aligned to special-token boundaries.
• Tokenize and apply the chat template once per request, then reuse the result across the KV index, routing policy, and P/D flow.
• Reduce JSON copies, repeated parsing, and intermediate string allocations in the request path.
• Optimize incremental streaming decode and stop-sequence detection.
• Evaluate an optional pre-tokenized/gRPC backend fast path.
• Add metrics for cache hits, CPU time, allocations, and Router-added latency.

Deliverables

• CPU fast-path benchmark suite.
• Request-path integration for the existing tokenizer and streaming modules.
• Bounded L0/L1 tokenizer cache.
• Request-scoped tokenization-result reuse.
• JSON and streaming hot-path optimizations.
• Design document or experimental implementation for an optional tokens-in/gRPC path.
• Before-and-after performance report.

Acceptance criteria

• Token IDs and chat-template results match the target vLLM version.
• Every cache has a capacity bound, TTL/eviction policy, and memory metrics; unbounded growth is not allowed.
• Publish CPU, memory, P50/P99 Router-added latency, and throughput results.
• Preserve existing HTTP behavior when the new fast path is disabled.
• Do not count the OpenTelemetry gRPC exporter as worker gRPC support.

Non-goals

• Complete MCP tool orchestration.
• Provider API translation.
• Reimplementing every Hugging Face multimodal processor inside the Router.

---

Issue 2: [Roadmap][P0] Add Event-driven Exact KV-aware Routing

Background

The current cache-aware policy maintains an approximate character-level tree from Router request history. It cannot determine which KV blocks are currently resident on a Worker or DP rank. Dynamo Router combines KV lifecycle events, a prefix index, and active load when selecting a Worker.

Scope

• Subscribe to vLLM KV Stored, Removed, and Cleared events.
• Build an exact KV index keyed by model, Worker, DP rank, and block key.
• Query request prefixes at token/block granularity and calculate actually reusable blocks.
• Combine uncached prefill, active decode, active requests, and KV transfer cost in a unified cost model.
• Automatically fall back to approximate cache-aware routing or P2C when KV Events are unavailable or the index is not trustworthy.
• Emit a structured route receipt and decision metrics.
• Validate block size, tokenizer compatibility, Worker epoch, and event freshness.

Deliverables

• vLLM KV Event subscriber.
• Token/block KV placement index.
• Unified KV-plus-load cost policy.
• Exact/approximate/load-aware fallback.
• Route receipt and Prometheus metrics.
• Comparative benchmark covering RR, P2C, approximate cache-aware, and exact KV-aware routing.

Acceptance criteria

• Stored, Removed, and AllBlocksCleared events update the index correctly.
• DP-rank-level KV placement matches actual Worker state.
• Requests continue through a fallback policy when events are unavailable.
• Report KV hit rate, TTFT P99, goodput/GPU, incorrect-routing rate, and Router overhead.
• Do not claim absolute superiority using competitor data collected in a different environment.

Non-goals

• A multi-data-center global index is not required in the first phase.
• Full multi-tier KV-offload orchestration is not required in the first phase.

---

Issue 3: [Roadmap][P1] Extend Session Affinity to Agent/Program-aware Scheduling

@warriorsniu

Background

Agent workflows contain multiple LLM calls, tool calls, and waiting periods. Per-request scheduling loses program-level context, which can cause session KV eviction, migration of one agent across Workers, and repeated prefill after a tool returns.

The current consistent-hash policy can read headers such as x-session-id, x-user-id, and x-tenant-id and attempts to map the same key to the same Worker. This provides sticky sessions, but it does not understand programs, steps, tool waits, KV TTL, or expected output length.

ThunderAgent introduces an LLM Program abstraction and a program-aware scheduler. Continuum uses a KV TTL based on recomputation/reload cost and eviction-induced queueing cost to selectively retain KV across tool calls.

Scope

• Define optional program/session/step metadata.
• Preserve existing x-session-id consistent-hash behavior as the baseline when program metadata is absent.
• Support program ID, step ID, agent role, priority, deadline, and expected output length.
• Select Workers using both KV affinity and live load at program scope.
• Add program-level fairness so long-running agent workflows cannot monopolize a Worker.
• Design a KV TTL/retention hint for tool-wait periods.
• Evaluate eviction or offload at TTL expiry and prefetch before a tool returns.
• Fall back to ordinary KV-aware routing when hints are absent or unsupported by the engine.
• Evaluate job completion time using real or public agent traces.

Deliverables

• Agent scheduling metadata contract.
• Program/session-aware policy evolved from the existing session-affinity policy.
• KV TTL/retention policy prototype.
• Optional prefetch/offload hint design.
• Agent trace replay and benchmark.

Acceptance criteria

• Ordinary requests do not need to supply agent metadata.
• Scheduling decisions for one program are explainable and replayable.
• Compare ordinary KV-aware routing, sticky sessions, and program-aware routing.
• Report agent job completion time, KV reuse, Worker migrations, and fairness.
• Router-to-engine KV control must receive explicit accept/reject feedback from the engine; the Router must not assume a hint was executed.

Non-goals

• The Router does not execute tools.
• The Router does not manage complete tool runtimes, disks, or network ports.
• Performance numbers reported by the research papers are not project results.

---

Issue 4: [Roadmap][P1] Extend P/D Routing to E/P/D

Background

A multimodal request can be decomposed into Encode, Prefill, and Decode stages. SGLang's EPD Router work introduces Encode Workers, a three-stage pipeline, and independent Worker policies. The referenced CI run covers its EPD routing pipeline work.

Scope

• Add an Encode Worker type, Encode pool, and service discovery.
• Support a minimal Encode → Prefill → Decode orchestration flow.
• Support independent policies for Encode, Prefill, and Decode.
• Carry encoder output and KV-transfer metadata between stages.
• Remain compatible with vLLM EC Connector/EPD interfaces rather than defining a separate private protocol.
• Skip Encode for text-only requests and continue using the existing P/D path.
• Add health, load, and latency metrics for all three stages.

Deliverables

• Encode Worker type and discovery.
• E/P/D pipeline MVP.
• Per-stage policy configuration.
• Text-only P/D fallback.
• End-to-end test with one multimodal model.

Acceptance criteria

• Encode, Prefill, and Decode can be deployed and scaled independently.
• Text-only requests do not enter the Encode stage.
• A failure in any stage produces an explicit error; execution must not continue with incomplete intermediate state.
• Streaming response and client-cancellation behavior are correct.

Non-goals

• A general-purpose Omni stage graph is not part of the first phase.
• Complex topology-aware EPD is not part of the first phase.

---

Issue 5: [Roadmap][P0/P1] Complete Multi-Router HA and State Recovery

Background

Exact KV state, sessions, and active load make the Router stateful. The current Router already has Worker health checks, retries, circuit breakers, liveness/readiness endpoints, and Axum graceful shutdown on SIGTERM or Ctrl+C. It does not have an explicit DRAINING state, readiness-based traffic removal, multi-Router state recovery, or cross-replica HA.

SMG uses SWIM and CRDTs to synchronize Worker, policy, rate-limit, and cache-aware state. llm-d supports leader election, warm standby, and constrained active-active operation. This issue completes multi-Router HA on top of the existing single-process reliability mechanisms rather than reimplementing health checks and retries.

Scope

• Extend the existing liveness/readiness endpoints and process-level graceful shutdown with an explicit DRAINING state, readiness-based traffic removal, and rolling-upgrade behavior.
• Add replay for KV Event gaps and a snapshot/rebuild path for cold starts.
• Provide a basic active-passive/warm-standby mode.
• Evaluate active-active operation in which exact KV indexes converge independently and only the minimum required state is synchronized.
• Define explicit EXACT, DEGRADED, and FALLBACK states during recovery.
• Add fault-injection, RTO, and state-convergence metrics.

Deliverables

• Graceful drain and readiness gate built on the existing shutdown/readiness implementation. @LOGO127
• KV-index replay/snapshot foundation.
• Warm-standby deployment example.
• Multi-Router recovery tests.
• Active-active design document as a stretch goal.

Acceptance criteria

• In a multi-Router deployment, restarting or rolling one Router does not make the service unavailable.
• A new Router does not perform exact KV routing until its index is trustworthy.
• Event gaps are detected, and failed recovery triggers safe fallback.
• Complete Router-kill, Worker-kill, event-loss, and 24-hour steady-state tests.

Non-goals

• Strong-consistency Raft is not a mandatory dependency in the first phase.
• The first phase does not synchronize every local counter or transient request state.

---

Dependencies

Issue 1: CPU Fast Path Integration ─┐
                                    ├─> Issue 3: Agent-aware Scheduling
Issue 2: Exact KV Routing ──────────┤
                 │                  └─> Issue 5: Router HA
                 │
                 └─> Issue 4: E/P/D reuses token/block and load foundations

References

CPU-side inference fast path

• SMG: The Case for Disaggregating CPU from GPU in LLM Serving

Exact KV-aware routing

• NVIDIA Dynamo Router Overview

Agent/program-aware scheduling

• ThunderAgent: A Simple, Fast and Program-Aware Agentic Inference System
• Continuum: Multi-Turn LLM Agent Scheduling with KV Cache TTL

E/P/D routing

• SGLang EPD Router CI Run
• SGLang EPD Router umbrella PR #15683
• vLLM EPD Tracker #52409

Router high availability

• SMG High Availability
• llm-d Router HA Modes

[wuhang2014]

We can include #236 in the roadmap as part of enhancing the router's high extensibility..

[LOGO127]

I'd be interested in contributing the drain/readiness foundation in section 5, rather than the whole HA workstream.

On current main 0519da5397f429af9e54d1e9601b0a135e338b65, src/server.rs delegates /readiness directly to the router, while the SIGTERM/Ctrl+C future returns directly into Axum's graceful shutdown. A possible first delivery is an opt-in, bounded pre-drain phase: keep the listener alive, make readiness unhealthy while preserving liveness, reject new inference admission, let already-admitted work finish, then enter the existing shutdown path. Default behavior would remain unchanged.

Before implementation, would this be a useful standalone scope, and is anyone already working on it? In particular, I'd like to agree on whether requests already accepted into the concurrency queue count as admitted, and which inference/transparent/PD routes must share the gate. I would test admission order and in-flight stream completion with real local TCP connections and controlled backends; that would not be presented as proof of multi-router availability, KV-state recovery, or the roadmap's 24-hour acceptance test.

AI assistance would be disclosed. This is a scope proposal based on the existing path, not a claim of an implemented fix or ownership of the broader roadmap.

[warriorsniu]

I’d like to take ownership of Issue 3: Extend Session Affinity to Agent/Program-aware Scheduling.

I have prior implementation and evaluation experience with program-level agent scheduling, including optional agent/program identity contracts, Program lifecycle tracking, TTL-based continuity protection, capacity-aware admission and resume, fairness safeguards, and inference-engine integration.

I propose to deliver this incrementally:

1. Define an optional program metadata contract while preserving the existing x-session-id behavior when program metadata is absent.
2. Extend session affinity into a program-aware Worker policy combining historical KV affinity with live load.
3. Add program-level fairness, decision metrics, and trace replay for reproducible evaluation.
4. Introduce a KV TTL/retention prototype behind an optional engine capability contract, with explicit accept/reject feedback.

I would start with a focused design document to align the Router/engine boundary, metadata compatibility, fallback behavior, and dependencies on exact KV observations before submitting implementation PRs.

[fake0fan]

Thanks for volunteering!

@warriorsniu, please open a focused RFC or sub-issue for the Agent/Program-aware Scheduling work so we can discuss the metadata contract, Router/engine boundary, fallback behavior, and incremental implementation there.

@LOGO127, please open a focused sub-issue for the Graceful Drain/Readiness foundation, including the admission boundary for queued requests, route coverage, and in-flight stream completion. This can stay scoped independently from the broader HA workstream.

Once the RFC/issues are open, we can link them back to this roadmap and assign the corresponding tasks there.

[jiannan-17]

hi @fake0fan
I'd like to pick up the L0 exact-match tokenizer cache in Issue 1. Draft issue: #269, any feedback would be appreciated. Thank you!

The first PR would cover a bounded cache with eviction, tests, and benchmark. L1 can follow later. Since the tokenizer on main doesn't cache results and isn't wired into the request path yet, I plan to start with a wrapper around the existing Tokenizer trait.

Proposed scope:

• Cache tokenization results for identical inputs, isolated by tokenizer/model identity and relevant encoding options.
• Add bounded capacity, an eviction/expiry policy, and cache hit/miss and memory metrics.
• Verify cached and uncached results are identical, with tests for eviction, concurrent access, and tokenizer isolation.
• Add focused benchmarks for hit/miss latency and memory usage.

[zjbpaul1317]

Hi @fake0fan,
I'd be interested in contributing to Issue 4: Extend P/D Routing to E/P/D.
My understanding is that the first goal should be a minimal, vLLM-compatible E/P/D routing path for multimodal requests, while preserving the existing P/D path for text-only requests. I would like to keep the initial scope focused on the Router orchestration and stage-level routing boundaries, rather than building a general-purpose workflow engine.
I propose to proceed incrementally:

1. Review the current P/D routing, DP-rank routing, KV connector, and worker discovery implementations.
2. Align the Router/engine contract with the existing vLLM EPD and EC Connector interfaces, including encoder output and KV-transfer metadata.
3. Define an Encode Worker type, Encode worker pool, health state, and service discovery model.
4. Add request classification:
   • text-only requests continue through the existing P/D path;
   • multimodal requests enter the E/P/D path.
5. Implement a minimal Encode -> Prefill -> Decode orchestration flow.
6. Add independent per-stage routing policies and basic health/load/latency metrics.
7. Add explicit failure and cancellation handling so that a failed stage cannot be followed by execution with incomplete intermediate state.
8. Add an end-to-end test with one multimodal model, while keeping the existing text-only P/D tests unchanged.
   The initial implementation would explicitly not include:

• a general-purpose stage graph or Omni workflow engine;
• complex topology-aware placement;
• a private replacement for the vLLM EC Connector or EPD protocol;
• broad multimodal processor reimplementation inside the Router;
• performance claims based only on external SGLang or Dynamo results.
  Before submitting the implementation, I would like to open a focused RFC or sub-issue to confirm:
• the Router/engine ownership boundary;
• the format and lifetime of encoder outputs;
• the KV-transfer metadata contract;
• retry and cancellation semantics across Encode, Prefill, and Decode;
• whether stage failures should fail the request or allow a bounded re-routing attempt;
• the current upstream dependency status of vLLM EPD support and tracker #52409.
  For the initial evaluation, I plan to compare:
• text-only P/D requests;
• multimodal E/P/D requests;
• Encode, Prefill, and Decode stage latency;
• stage-level failure behavior;
• client cancellation and streaming completion;
• independent scaling of the three worker pools.
  I will keep the existing HTTP behavior and text-only P/D routing compatible throughout the work.

[fake0fan]

hi @fake0fan I'd like to pick up the L0 exact-match tokenizer cache in Issue 1. Draft issue: #269, any feedback would be appreciated. Thank you!

The first PR would cover a bounded cache with eviction, tests, and benchmark. L1 can follow later. Since the tokenizer on main doesn't cache results and isn't wired into the request path yet, I plan to start with a wrapper around the existing Tokenizer trait.

Proposed scope:

• Cache tokenization results for identical inputs, isolated by tokenizer/model identity and relevant encoding options.
• Add bounded capacity, an eviction/expiry policy, and cache hit/miss and memory metrics.
• Verify cached and uncached results are identical, with tests for eviction, concurrent access, and tokenizer isolation.
• Add focused benchmarks for hit/miss latency and memory usage.

Thanks @jiannan-17! The proposed scope looks reasonable to me. Starting with a bounded L0 exact-match cache and leaving L1 and request-path integration for follow-up work makes sense. Looking forward to your contribution!

[fake0fan]

Thanks @zjbpaul1317 for the thoughtful proposal! The overall direction sounds well aligned with the E/P/D roadmap item.

There is already an open draft PR, #266, that adds E+PD and E/P/D orchestration with JSON metadata. Could you please take a look and clarify how your proposed work differs from or complements that PR? It would be helpful to understand whether you plan to build on #266, address its remaining gaps, or pursue a different design.

Thanks again, and looking forward to your contribution!