[Roadmap] Rust Frontend Feature Parity

[BugenZhao]

Motivation

The Rust frontend has landed in the vLLM repository as a drop-in alternative to the Python API server process. It is enabled through VLLM_USE_RUST_FRONTEND=1 and uses the existing vLLM engine/core boundary.

The Rust frontend is still experimental and not yet feature-complete relative to the Python frontend. This issue tracks the roadmap for the remaining gaps in feature parity and provides community contributors with a place to find scoped work.

This roadmap prioritizes user value and the Rust frontend's architectural goals over blanket 1:1 parity with the Python frontend. In particular, we may skip:

• Features / endpoints / options that are not commonly used in production
• Options that mainly reflect Python frontend implementation details or Python-side workarounds
• Features whose Python implementation has accumulated enough complexity that redesigning or removing them may be better than porting them directly

For items marked with 🧭: Please start with a design discussion before implementation, covering the intended API/architecture, co-design with the engine if needed, and the testing plan.

Related context:

• Original RFC: [RFC]: Rust front-end #40846
• Integration PRs
  • [Frontend][RFC] Rust front-end integration #40848
  • [Rust Frontend] Move code from vllm-frontend-rs #43283

Current status

• Rust code lives in the main vLLM repository under rust/
• vllm-rs integrates with vllm serve as a Python-supervised frontend subprocess
• Core /v1/chat/completions support (streaming & non-streaming)
• Core /v1/completions support (streaming & non-streaming)
• Tool calling and reasoning output support for several key model families
• Commonly used OpenAI-compatible sampling options
• Image-only multimodal chat path for registered image processor/model specs
• Multi-engine internal load balancing with scheduler-stat-aware routing
• Admin and operational routes: health, load, version, metrics, reset-cache, sleep/wake
• Mock-engine and HTTP integration-test infrastructure for Rust frontend development

Roadmap

Distributed serving and large-scale control plane

• External DP load-balancing mode: support the Python-style shape where each API server connects to one DP rank 🧭
• Hybrid DP load-balancing mode: support per-node internal LB plus cross-node external LB 🧭
• Elastic EP support: implement /scale_elastic_ep, /is_scaling_elastic_ep, request draining, and scaling-state rejection behavior 🧭

Additional APIs

• Anthropic Messages API: /v1/messages and /v1/messages/count_tokens 🧭
• Non-stateful Responses API: start with POST /v1/responses without requiring durable response storage 🧭
• Pooling / embedding APIs: /v1/embeddings, /pooling, /classify, /score, /v1/score, /rerank, /v1/rerank, /v2/rerank 🧭
• Speech-to-text APIs: /v1/audio/transcriptions and /v1/audio/translations 🧭
• Realtime API: WebSocket /v1/realtime 🧭
• Generative scoring API: /generative_scoring
• Tokenization APIs: /tokenize, /detokenize, and /tokenizer_info
• Extend the Rust gRPC interface to cover embeddings and other non-generation outputs 🧭

Request compatibility and validation

• Audit request parameter coverage and validation, making unsupported features fail clearly
• stream_options.continuous_usage_stats
• n > 1 for chat completions and completions 🧭
• Beam search via the existing use_beam_search request parameter, including length_penalty
• truncate_prompt_tokens
• thinking_token_budget
• include_reasoning=false
• repetition_detection
• echo behavior for token-ID completions prompts
• max_tokens=0 prompt-only behavior
• Request ID handling in requests and responses
• Tracing 🧭

Production readiness

• TLS / SSL support: --ssl-keyfile, --ssl-certfile, and related serving arguments
• API key authentication: --api-key / VLLM_API_KEY
• CORS middleware and CLI argument parity
• Reverse-proxy --root-path support
• /server_info route parity
• Logging argument parity: max log length, output logging, delta logging, access-log exclusions, and log-level controls

LoRA adapter support

• Startup LoRA adapter loading through --lora-modules
• Runtime adapter loading/unloading: /v1/load_lora_adapter and /v1/unload_lora_adapter
• Multiple-adapter request routing and validation

Multimodal support

• image_url chat content path for the currently registered image model specs
• Image embedding inputs
• Audio inputs and audio embedding inputs 🧭
• Video inputs and video chunk inputs 🧭
• mm_processor_kwargs and media_io_kwargs
• Broader model-family coverage beyond the current image-only processor registry 🧭

RL / admin / lifecycle APIs

• /pause, /resume, and /is_paused
• /abort_requests
• /update_weights and /init_weight_transfer_engine
• /get_world_size

Tool and reasoning parser parity

• parallel_tool_calls
• Tool-choice constraints: required and named function 🧭
• Tool-choice constraints: allowed_tools 🧭
• Add missing tool and reasoning parsers for currently unsupported Python parser names.
• Add model-family roundtrip fixtures for newly added parsers

Parser work should prioritize production use and key model families over 100% parity. Additionally, the Rust frontend has a redesigned tool and reasoning parser architecture for more efficient, robust, and maintainable parsing, so prefer adapting the current Rust design over line-by-line Python ports.

Full parser parity checklist

Tool parsers

• apertus
• cohere_command3
• cohere_command4
• ernie45
• functiongemma
• gigachat3
• granite
• granite-20b-fc
• granite4
• hunyuan_a13b
• internlm
• jamba
• lfm2
• llama4_pythonic 🧭
• longcat
• mimo
• minimax
• minimax_m2
• minimax_m3
• minicpm5
• olmo3
• phi4_mini_json
• poolside_v1
• pythonic 🧭
• seed_oss
• step3
• step3p5
• xlam

Reasoning parsers

• cohere_command3
• cohere_command4
• ernie45
• granite
• holo2
• hunyuan_a13b
• hy_v3
• mimo
• minimax_m2_append_think
• mistral
• olmo3
• poolside_v1
• seed_oss
• step3p5

Testing and compatibility infrastructure

• Improve E2E coverage through real vllm serve with VLLM_USE_RUST_FRONTEND=1 on CI
• Documentation for building, testing, and contributing to the Rust frontend

[snailwei]

not working with 0.22.0

(EngineCore pid=739) INFO 06-04 11:58:06 [vllm.py:977] Asynchronous scheduling is enabled.
(EngineCore pid=739) INFO 06-04 11:58:06 [kernel.py:270] Final IR op priority after setting platform defaults: IrOpPriorityConfig(rms_norm=['native'], fused_add_rms_norm=['native'])
ERROR 06-04 11:58:06 [utils.py:546] Exception occurred while running API servers: Process RustFrontend (PID: 740) died with exit code 2
ERROR 06-04 11:58:06 [utils.py:546] Traceback (most recent call last):
ERROR 06-04 11:58:06 [utils.py:546]   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/utils.py", line 533, in wait_for_completion_or_failure
ERROR 06-04 11:58:06 [utils.py:546]     raise RuntimeError(
ERROR 06-04 11:58:06 [utils.py:546] RuntimeError: Process RustFrontend (PID: 740) died with exit code 2
(Worker_TP0 pid=940) INFO 06-04 11:58:07 [multiproc_executor.py:775] Parent process exited, terminating worker queues
(Worker_TP1 pid=941) INFO 06-04 11:58:07 [multiproc_executor.py:872] WorkerProc shutting down.
(Worker_TP0 pid=940) INFO 06-04 11:58:07 [multiproc_executor.py:872] WorkerProc shutting down.
Traceback (most recent call last):
  File "/usr/local/bin/vllm", line 10, in <module>
    sys.exit(main())
             ^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/vllm/entrypoints/cli/main.py", line 92, in main
    args.dispatch_function(args)
  File "/usr/local/lib/python3.12/dist-packages/vllm/entrypoints/cli/serve.py", line 144, in cmd
    run_multi_api_server(args)
  File "/usr/local/lib/python3.12/dist-packages/vllm/entrypoints/cli/serve.py", line 359, in run_multi_api_server
    wait_for_completion_or_failure(
  File "/usr/local/lib/python3.12/dist-packages/vllm/v1/utils.py", line 533, in wait_for_completion_or_failure
    raise RuntimeError(
RuntimeError: Process RustFrontend (PID: 740) died with exit code 2