Skip to content

feat: route chat template by model_type and support DeepSeek V4 cpp template.#74

Merged
RobbieLeung merged 7 commits into
xLLM-AI:mainfrom
phantomlei3:feat/support-v4-chat-template
Jun 25, 2026
Merged

feat: route chat template by model_type and support DeepSeek V4 cpp template.#74
RobbieLeung merged 7 commits into
xLLM-AI:mainfrom
phantomlei3:feat/support-v4-chat-template

Conversation

@phantomlei3

@phantomlei3 phantomlei3 commented Jun 24, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor
  1. 抽象化重构：抽出 ChatTemplate 基类，把原本硬编码的 Jinja 模板链路解耦，并新增 encode_add_special_tokens() 钩子统一控制编码时是否追加 special token。
  2. 新增 DeepSeek V4 支持：新增 DeepseekV4CppChatTemplate 包装上游 C++ 模板（special token 设为不追加以避免双 BOS），并通过 config.json 的 model_type 自动路由选择模板。
  3. 配套改造：为 Tokenizer::encode() 增加 add_special_tokens 参数（不支持的 tokenizer 告警提示双 BOS 风险），并拆出 torch-free 的 mm_content_text工具函数供复用。

phantomlei3 and others added 7 commits June 24, 2026 15:34
@phantomlei3 @claude
refactor: add ChatTemplate abstraction and tokenizer add_special_tokens
ea758a2 
Introduce xllm_service::ChatTemplate abstract base with apply() and
encode_add_special_tokens(); make JinjaChatTemplate derive from it
(returns true). Add a bool add_special_tokens=true parameter to
Tokenizer::encode and plumb it through FastTokenizer (replacing the
hardcoded 1); SentencePiece/Tiktoken accept it to unify the signature
while keeping current behavior. Scheduler now holds unique_ptr<ChatTemplate>
and passes encode_add_special_tokens() to encode. Pure enabling scaffolding
for DeepSeek V4 C++ chat template; zero behavior change on existing paths.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@phantomlei3 @claude
feat: add DeepseekV4CppChatTemplate wrapper reusing upstream V4 cpp t…
6ad866c 
…emplate

Wrap xllm::DeepseekV4CppTemplate behind the xllm_service::ChatTemplate
abstraction to render DeepSeek V4 prompts. The wrapper projects service-side
Message / TokenizerArgs / JsonTool onto the upstream xllm types inside the .cpp,
confining all xllm/torch contact to one TU; encode_add_special_tokens() returns
false since the V4 template emits BOS and role markers literally.

- New deepseek_v4_cpp_chat_template.{h,cpp}: projection + apply, MMContentVec
  flattened to text, empty tool_call_id projected as unset optional.
- Extract torch-free flat_text / needs_content_vec into shared mm_content_text
  header reused by message_projection and the V4 wrapper.
- CMake compiles the upstream deepseek_v4_cpp_template.cpp with torch header
  include paths only; torch stays header-only, libtorch is never linked.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@phantomlei3 @claude
feat: route DeepSeek V4 chat template by config.json model_type
0cc56b4 
Read model_type from the model directory's config.json at Scheduler
construction (non-vllm backends only) and select the chat template:
deepseek_v4 -> DeepseekV4CppChatTemplate (encode_add_special_tokens
false, avoiding a double BOS), everything else (including missing or
unknown model_type) -> JinjaChatTemplate. Log the chosen template and
model_type at startup.

Selection logic lives in chat_template/model_type.{h,cpp} as pure,
path-injectable functions (select_chat_template_kind, load_model_type).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@phantomlei3 @claude
test: add unit tests for ChatTemplate, V4 cpp template and model_type…
cd27283 
… routing

Collect the unit tests previously bundled into the three feature commits
into a single test-only commit, keeping production code and tests separate:

- tokenizer/tokenizer_test.cpp + cc_test target (encode add_special_tokens)
- chat_template/jinja_chat_template_test.cpp additions
- chat_template/deepseek_v4_cpp_chat_template_test.cpp + cc_test target
- chat_template/model_type_test.cpp + cc_test target

No production code changes; only test sources and their cc_test wiring.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@phantomlei3 @claude
docs: condense verbose comments in V4 chat template changes
fa7dbdb 
Trim the redundant comments added across the V4 chat template work to
the non-obvious "why", dropping restated signatures and wording:

- chat_template / deepseek_v4_cpp_chat_template / mm_content_text /
  model_type headers: shorten class and function doc comments.
- deepseek_v4_cpp_chat_template.cpp: tighten the torch/glog LOG macro,
  TokenizerArgs projection, and content-flatten notes.
- scheduler.cpp + CMakeLists: collapse multi-line notes to one or two.
- tests: shorten a few over-long intent comments.

Comment-only change; no behavior change.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@phantomlei3 @claude
fix: warn on ignored add_special_tokens and guard chat_template null
9b24cfd 
SentencePiece/Tiktoken tokenizers cannot suppress special tokens, so they
now emit a one-shot warning when encode is called with add_special_tokens
=false, surfacing the double-BOS risk the V4 cpp template relies on avoiding.

Also guard chat_template_ before dereferencing it in the encode branch of
Scheduler::schedule, matching the existing null check in the apply branch.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@phantomlei3 @claude
style: apply clang-format to V4 chat template changes
5e19ad7 
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
gemini-code-assist[bot]
gemini-code-assist Bot reviewed Jun 24, 2026
View reviewed changes

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request introduces support for the DeepSeek V4 C++ chat template by defining a new ChatTemplate base interface and implementing DeepseekV4CppChatTemplate. The Scheduler is updated to dynamically select the appropriate template based on the model's config.json. Additionally, the Tokenizer::encode interface is updated to accept an add_special_tokens parameter to prevent duplicate BOS tokens. The reviewer feedback suggests using python3 instead of python in CMake to locate PyTorch headers, ensuring compatibility on modern Linux distributions.

Important

The consumer version of Gemini Code Assist on GitHub is being sunset. Starting June 18, 2026, new organization installations will be blocked, and all code review activity will officially cease on July 17, 2026.
For more details on the timeline and next steps, please review the Help Documentation.

Comment thread xllm_service/chat_template/CMakeLists.txt
@RobbieLeung
RobbieLeung merged commit a3a41de into xLLM-AI:main Jun 25, 2026
1 check passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@RobbieLeung RobbieLeung RobbieLeung approved these changes
@XuZhang99 XuZhang99 XuZhang99 approved these changes
+1 more reviewer
@gemini-code-assist gemini-code-assist[bot] gemini-code-assist[bot] left review comments
Reviewers whose approvals may not affect merge requirements

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@phantomlei3 @RobbieLeung @XuZhang99