fix(tokenizer): Kimi-K3 prompt segments and skip_special_tokens in the tiktoken backend - #2421

Merged

CatherineSue merged 3 commits into

smg-project:mainfrom

syuoni:fix/k3-segment-rendering

Sep 5, 2026

Commits on Sep 5, 2026

fix(tokenizer): honor skip_special_tokens in the tiktoken decoder
syuoni
committed
79a63d8
fix(tokenizer): encode Kimi-K3 prompts per segment like the reference renderer
syuoni
committed
9a49a96
fix(tokenizer): skip built-in special tokens and keep flat prefixes in one segment
syuoni
committed
95f68ec