Qwen 3.5 lexer support and close the HF normalizer parity gap - #387

Draft

kylegrover wants to merge 11 commits into

zspacelabs:mainfrom

kylegrover:kyleg/qwen3.5-lexer

Commits on May 11, 2026

Add HuggingFace pretrained vocab provider and refactor bench data iteration
crutcher
committed
4c0a399
Extend HF vocab provider to support byte-level BPE tokenizers
crutcher
committed
643dec6
Make hf factory a better pass-through factory.
crutcher
committed
366dd4e
Add `VocabQuery::to_context()` method and refactor HF vocab provider
crutcher
committed
7ed89a8
Partial work on Qwen3.5.
crutcher
committed
b0e9475
Update partial support for HF with basic errors.
crutcher
committed
460dacf
Fix no-std build.
crutcher
committed
acf7aef