Skip to content

fix(tokenizer): Kimi-K3 prompt segments and skip_special_tokens in the tiktoken backend - #2421

Merged
CatherineSue merged 3 commits into
smg-project:mainfrom
syuoni:fix/k3-segment-rendering
Sep 5, 2026
Merged

fix(tokenizer): Kimi-K3 prompt segments and skip_special_tokens in the tiktoken backend#2421
CatherineSue merged 3 commits into
smg-project:mainfrom
syuoni:fix/k3-segment-rendering

fix(tokenizer): skip built-in special tokens and keep flat prefixes i…

95f68ec
Select commit
Loading
Failed to load commit list.
DCO / DCO succeeded Sep 5, 2026 in 0s

DCO

All commits are signed off!

View more details on DCO