Skip to content

feat(deepseek-v2-lite): add mixed-request serving gate with per-request KV ownership - #441

Merged
xiaguan merged 2 commits into
pegainfer-project:mainfrom
CAICAIIs:feat/dsv2lite-mixed-serving-admission
Jun 23, 2026
Merged

feat(deepseek-v2-lite): add mixed-request serving gate with per-request KV ownership#441
xiaguan merged 2 commits into
pegainfer-project:mainfrom
CAICAIIs:feat/dsv2lite-mixed-serving-admission

Conversation

@CAICAIIs

Copy link
Copy Markdown
Collaborator

Summary

This PR adds the first greedy mixed-request serving gate for DeepSeek-V2-Lite EP2.

  • add MixedRequestScheduler for explicit admit / reject / defer handling
  • keep one owned DecodeCache per active request
  • retire requests independently on EOS, length, rejection, error, or sink failure
  • batch decode only when active requests share the same decode position; otherwise fall back to single-row decode
  • extend the EP2 E2E to cover same-position batching, single-row fallback, and invalid-request isolation
  • update the DeepSeek-V2-Lite status docs to keep the claim boundary explicit

Validation

  • cargo fmt --all --check
  • git diff --check
  • cargo test --offline --release -p openinfer-deepseek-v2-lite --features deepseek-v2-lite --lib -- --nocapture
  • host-staged EP2 E2E
  • NCCL EP2 E2E with an updated NCCL runtime
  • HF / host-staged / NCCL exact compare

Result: all_token_text_exact

Scope

Not claimed here: vLLM parity, production EP readiness, sparse dispatch, or HTTP continuous batching.

Closes #281

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: ca281080ea

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

Comment on lines +262 to +266
let original_caches: Vec<_> = self
.active
.iter()
.map(|state| state.cache.clone())
.collect();

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Avoid cloning full KV caches on each batched decode

When two or more active requests share a decode position, this snapshots every active DecodeCache before every generated token just to support fallback. These caches are the full f32 per-layer KV history and grow with context; for DeepSeek-V2-Lite that is hundreds of MB per token for modest 8-row batches, so the intended same-position mixed-serving path can be dominated by host copies or hit memory pressure even when batched decode succeeds. Avoid taking full-cache clones on the hot path, or make fallback use a smaller rollback strategy.

Useful? React with 👍 / 👎.

@xiaguan
xiaguan merged commit a296434 into pegainfer-project:main Jun 23, 2026
1 check passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

dsv2lite: add mixed-request serving admission and KV ownership

2 participants