feat(deepseek-v2-lite): add mixed-request serving gate with per-request KV ownership - #441
Conversation
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: ca281080ea
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
| let original_caches: Vec<_> = self | ||
| .active | ||
| .iter() | ||
| .map(|state| state.cache.clone()) | ||
| .collect(); |
There was a problem hiding this comment.
Avoid cloning full KV caches on each batched decode
When two or more active requests share a decode position, this snapshots every active DecodeCache before every generated token just to support fallback. These caches are the full f32 per-layer KV history and grow with context; for DeepSeek-V2-Lite that is hundreds of MB per token for modest 8-row batches, so the intended same-position mixed-serving path can be dominated by host copies or hit memory pressure even when batched decode succeeds. Avoid taking full-cache clones on the hot path, or make fallback use a smaller rollback strategy.
Useful? React with 👍 / 👎.
Summary
This PR adds the first greedy mixed-request serving gate for DeepSeek-V2-Lite EP2.
MixedRequestSchedulerfor explicit admit / reject / defer handlingDecodeCacheper active requestValidation
cargo fmt --all --checkgit diff --checkcargo test --offline --release -p openinfer-deepseek-v2-lite --features deepseek-v2-lite --lib -- --nocaptureResult:
all_token_text_exactScope
Not claimed here: vLLM parity, production EP readiness, sparse dispatch, or HTTP continuous batching.
Closes #281