[Router] Add WASM pluggable OnRequest middleware (v0.1) by spencerr221 · Pull Request #251 · vllm-project/router

spencerr221 · 2026-09-09T02:35:45Z

Introduce the RFC HTTP envelope WIT, a Wasmtime host runtime with digest pinning and fail-closed errors, CLI/Python wiring, and an example guest under examples/wasm_middleware. Default attach point is POST /v1/chat/completions, with optional --wasm-middleware-route.

Summary

Add vllm:router-middleware@0.1.0 HTTP envelope WIT and Wasmtime host runtime
Wire --wasm-middleware / --wasm-middleware-sha256 / --wasm-middleware-route
Default attach point: POST /v1/chat/completions (fail-closed, opt-in)
Example guest under examples/wasm_middleware/
Refs: [RFC]: WASM-Based Pluggable Middleware for vLLM Router #236

Test plan

examples/wasm_middleware/build.sh
cargo test --lib wasm_middleware::
cargo test --bin vllm-router parses_wasm_middleware
CI WASM Middleware workflow

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: c9f2aeb2f2

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

chatgpt-codex-connector · 2026-09-09T02:39:17Z

+        let mut linker = Linker::new(&engine);
+        wasmtime_wasi::p2::add_to_linker_sync(&mut linker)
+            .map_err(|err| WasmMiddlewareError::Engine(err.to_string()))?;


Enforce deadlines across synchronous WASI host calls

A component can call wasi:clocks/monotonic-clock.subscribe followed by blocking wasi:io/poll.poll; because the full synchronous WASI implementation is linked here, that host call can block a worker far beyond the configured 100 ms epoch deadline. Epoch interruption only regains control when execution returns to Wasm, so a plugin that sleeps indefinitely can occupy every worker and leave subsequent matching requests permanently rejected or queued. Use async WASI execution with an outer timeout/cancellation mechanism, or avoid exposing blocking imports, so the advertised execution deadline covers host calls too.

Useful? React with 👍 / 👎.

chatgpt-codex-connector · 2026-09-09T02:39:17Z

+    if let Some(runtime) = wasm_runtime {
+        protected_routes = protected_routes.route_layer(axum::middleware::from_fn_with_state(
+            crate::wasm_middleware::WasmRouteMiddlewareState {
+                runtime,
+                max_payload_size,
+            },
+            crate::wasm_middleware::wasm_on_request_middleware,
        ));


Apply middleware to every configurable server route

When --wasm-middleware-route names an admin, public, worker-management, or transparent-proxy path (for example /add_worker or a backend-specific fallback path), the plugin is never invoked because this layer is attached only to protected_routes before the other routers and fallback are merged. The CLI describes the option as an arbitrary HTTP path and accepts these values without validation, so operators can believe a configured request policy is active when it is silently bypassed. Attach the layer at the combined application level, or reject routes outside the supported protected-route set.

Useful? React with 👍 / 👎.

spencerr221 · 2026-09-10T02:28:50Z

e2e integration tests passed：

wuhang2014 · 2026-09-10T08:34:17Z

Please follow the template(https://github.com/vllm-project/router/blob/main/.github/PULL_REQUEST_TEMPLATE.md) for PR's body.

Copilot

🟡 Changes recommended

Security exposure, ineffective resource limits, route bypasses, and missing observability must be addressed.

Once you've addressed the issues Copilot identified, you can request another Copilot review.

Pull request overview

Adds opt-in WASM OnRequest middleware with a versioned WIT contract, Wasmtime runtime, configuration wiring, example plugin, and CI coverage.

Changes:

Implements bounded, digest-pinned WASM request processing.
Adds Rust/Python CLI configuration and route attachment.
Adds an example component and integration workflow.

File summaries

File	Description
`wit/router-middleware.wit`	Defines the middleware ABI.
`src/wasm_middleware.rs`	Implements the host runtime and Axum adapter.
`src/server.rs`	Loads and mounts middleware.
`src/main.rs`	Adds Rust CLI options.
`src/lib.rs`	Adds Python binding fields.
`README.md`	Documents usage.
`py_src/vllm_router/router_args.py`	Adds Python CLI options.
`py_test/integration/test_wasm_middleware.py`	Tests end-to-end behavior.
`py_test/fixtures/router_manager.py`	Supports binary and WASM flags.
`py_test/fixtures/mock_worker.py`	Exposes forwarded headers.
`examples/wasm_middleware/src/lib.rs`	Implements the example guest.
`examples/wasm_middleware/README.md`	Documents the example.
`examples/wasm_middleware/Cargo.toml`	Configures the guest crate.
`examples/wasm_middleware/build.sh`	Builds the guest component.
`examples/wasm_middleware/.gitignore`	Excludes generated artifacts.
`Cargo.toml`	Adds WASM runtime dependencies.
`Cargo.lock`	Locks new dependencies.
`.github/workflows/wasm-middleware.yml`	Builds and tests middleware.

Review details

Files reviewed: 17/18 changed files
Comments generated: 10
Review effort level: Balanced

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

+    headers
+        .iter()
+        .map(|(name, value)| WitHeader {
+            name: name.as_str().to_string(),
+            value: value.as_bytes().to_vec(),


+        protected_routes = protected_routes.route_layer(axum::middleware::from_fn_with_state(
+            crate::wasm_middleware::WasmRouteMiddlewareState {
+                runtime,
+                max_payload_size,
+            },


+        let mut engine_config = Config::new();
+        engine_config.wasm_component_model(true);
+        engine_config.epoch_interruption(true);
+        let mut pooling = PoolingAllocationConfig::default();


+        }
+
+        let route_set = config.routes.iter().cloned().collect();
+        let queue_permits = Arc::new(Semaphore::new(config.queue_capacity));


+            if status < 400 {
+                return Err(WasmMiddlewareError::InvalidAction(format!(
+                    "reject status must be >= 400, got {status}"
+                )));
+            }


+        if let Ok(header_name) = HeaderName::from_bytes(name.as_bytes()) {
+            headers.remove(header_name);
+        }


+
+    let method = request.method().as_str().to_string();
+    let query = request.uri().query().unwrap_or_default().to_string();
+    let request_id = request_id_from_headers(request.headers());


+    let (mut parts, body) = request.into_parts();
+    let bytes = match axum::body::to_bytes(body, state.max_payload_size).await {
+        Ok(bytes) => bytes,
+        Err(err) => {
+            let message = err.to_string();
+            if message.contains("length limit exceeded") {
+                warn!(
+                    "wasm middleware rejected oversized body (limit {} bytes)",
+                    state.max_payload_size
+                );


+        Err(err) => {
+            warn!("wasm middleware failed closed: {err}");
+            StatusCode::INTERNAL_SERVER_ERROR.into_response()


+        self.inner.route_set.contains(path)
+    }
+
+    pub fn metrics(&self) -> WasmRuntimeMetrics {


Introduce the RFC HTTP envelope WIT, a Wasmtime host runtime with digest pinning and fail-closed errors, CLI/Python wiring, and an example guest under examples/wasm_middleware. Default attach point is POST /v1/chat/completions, with optional --wasm-middleware-route. Signed-off-by: LiuBingyu <liubingyu4@huawei.com>

Exercise modify/reject/path-isolation against a real Router binary and mock worker, and extend CI to build the binary and run the pytest. Signed-off-by: LiuBingyu <liubingyu4@huawei.com>

Ignore wit/wast in codespell, and COPY wit/ into Dockerfile.router so image builds (Trivy) can compile wasm_middleware bindgen.

python -m build packs from the sdist; without wit/, wasmtime bindgen fails in the Buildkite Build Python Wheel job.

wuhang2014

It looks good for now except no integration test in Rust in tests/ which can be run by cargo test --test '*' with mock HTTP workers in tests/common/.

Rust wasm_middleware unit tests and the Python integration test need the example component artifact that build.sh produces.

chatgpt-codex-connector Bot reviewed Sep 9, 2026

View reviewed changes

wuhang2014 requested a balanced review from Copilot September 10, 2026 08:31

Copilot started reviewing on behalf of wuhang2014 September 10, 2026 08:39 View session

Copilot AI reviewed Sep 10, 2026

View reviewed changes

LiuBingyu added 3 commits September 14, 2026 10:47

[Router] Add WASM middleware integration test with mock worker

ed7734b

Exercise modify/reject/path-isolation against a real Router binary and mock worker, and extend CI to build the binary and run the pytest. Signed-off-by: LiuBingyu <liubingyu4@huawei.com>

[CI] Fix codespell false positives and Docker WIT copy

d3dfefc

Ignore wit/wast in codespell, and COPY wit/ into Dockerfile.router so image builds (Trivy) can compile wasm_middleware bindgen.

spencerr221 force-pushed the feat/wasm-pluggable-middleware branch from 2388b0b to d3dfefc Compare September 14, 2026 02:47

[CI] Include wit/ in sdist for Python wheel builds

8744f97

python -m build packs from the sdist; without wit/, wasmtime bindgen fails in the Buildkite Build Python Wheel job.

hsliuustc0106 added the enhancement New feature or request label Sep 14, 2026

wuhang2014 requested changes Sep 14, 2026

View reviewed changes

[CI] Build example WASM plugin before router unit/Python tests

24924c1

Rust wasm_middleware unit tests and the Python integration test need the example component artifact that build.sh produces.

Navigation Menu

Uh oh!

[Router] Add WASM pluggable OnRequest middleware (v0.1) - #251

[Router] Add WASM pluggable OnRequest middleware (v0.1)#251
spencerr221 wants to merge 5 commits into
vllm-project:mainfrom
spencerr221:feat/wasm-pluggable-middleware

spencerr221 commented Sep 9, 2026 •

edited

Loading

Uh oh!

chatgpt-codex-connector Bot left a comment

Uh oh!

chatgpt-codex-connector Bot Sep 9, 2026

Uh oh!

chatgpt-codex-connector Bot Sep 9, 2026

Uh oh!

spencerr221 commented Sep 10, 2026

Uh oh!

wuhang2014 commented Sep 10, 2026

Uh oh!

Copilot AI left a comment

Uh oh!

wuhang2014 left a comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Uh oh!

Conversation

spencerr221 commented Sep 9, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Test plan

Uh oh!

chatgpt-codex-connector Bot left a comment

Choose a reason for hiding this comment

💡 Codex Review

Uh oh!

chatgpt-codex-connector Bot Sep 9, 2026

Choose a reason for hiding this comment

Uh oh!

chatgpt-codex-connector Bot Sep 9, 2026

Choose a reason for hiding this comment

Uh oh!

spencerr221 commented Sep 10, 2026

Uh oh!

wuhang2014 commented Sep 10, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

🟡 Changes recommended

Uh oh!

wuhang2014 left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

spencerr221 commented Sep 9, 2026 •

edited

Loading