Skip to content

Qwen 3.5 lexer support and close the HF normalizer parity gap - #387

Draft
kylegrover wants to merge 11 commits into
zspacelabs:mainfrom
kylegrover:kyleg/qwen3.5-lexer
Draft

Qwen 3.5 lexer support and close the HF normalizer parity gap#387
kylegrover wants to merge 11 commits into
zspacelabs:mainfrom
kylegrover:kyleg/qwen3.5-lexer

Conversation

@kylegrover

@kylegrover kylegrover commented May 12, 2026

Copy link
Copy Markdown

Completes Qwen 3.5 tokenizer integration and closes remaining Hugging Face parity gap on normalized Unicode input.

  • adds the Qwen-specific lexer and supporting pair-vocab groundwork
  • wires the regex-automata transform registration into the existing inventory-based integration path
  • fixes the remaining HF normalizer mismatch for decomposed Unicode input
  • adds a dedicated Qwen encode benchmark against tokenizers
  • cleans up the normalization API so normalization is owned by the vocab/encoder path rather than the raw spanner config API

What Changed

  • grounded BPE pair-vocab construction and added the Qwen-specific Logos lexer
  • registered regex-automata transforms via inventory hooks so the Qwen lexer plugs into the existing loader path cleanly
  • relaxed unified vocab validation so pair-vocab tokens only need to be a subset of span-vocab tokens, which matches the actual Qwen vocabulary shape
  • added internal Unicode normalization support for the HF loader path
  • parsed supported HF normalizers (NFC, NFD, NFKC, NFKD, and sequences) and preserved AddedToken.normalized behavior when loading special tokens
  • moved input normalization ownership to UnifiedTokenVocab so encoder behavior is correct without implying that direct TextSpanner consumers operate on normalized text
  • added qwen_encoding_single to wordchipper-bench for repeatable wordchipper vs HF throughput checks

Validation

  • cargo test -p wordchipper --features client normalizer
  • cargo test -p wordchipper --features client qwen
  • cargo bench -p wordchipper-bench --bench qwen_encoding_single -- --sample-count 10

Benchmark

Final qwen_encoding_single results on this branch with --sample-count 10:

  • diverse: wordchipper 36.09 MB/s, tokenizers 2.345 MB/s, 15.39x faster
  • english: wordchipper 36.02 MB/s, tokenizers 2.026 MB/s, 17.78x faster

Notes

  • This closes the previously observed decomposed-Unicode mismatch against the Qwen HF tokenizer.
  • The normalizer follow-up keeps the public API consistent by making normalization an encoding concern instead of a raw spanner concern.

crutcher and others added 11 commits May 10, 2026 23:10
@crutcher
Add HuggingFace pretrained vocab provider and refactor bench data ite…
4c0a399 
…ration

- Add `huggingface` feature and `HFVocabProvider` for loading tokenizers from HuggingFace Hub
- Refactor bench data methods to use `.values()` instead of `.iter()` for cleaner iteration
- Fix clippy warning by allowing explicit counter loop in datagym vocab.

This is broken still:
```
crutcher@HeatLamp:~/git/wordchipper$ echo 'hello world' | cargo run -p wordchipper-cli -- cat --model 'hf:Xenova/gpt2' --encode
    Finished `dev` profile [unoptimized + debuginfo] target(s) in 0.16s
     Running `target/debug/wordchipper-cli cat --model 'hf:Xenova/gpt2' --encode`
Error: External("No pre-tokenizer")
```
@crutcher
Extend HF vocab provider to support byte-level BPE tokenizers
643dec6 
- Add support for ByteLevel and Sequence pre-tokenizers in addition to Split
- Implement proper byte-to-unicode mapping for byte-level BPE vocabularies
- Filter out special tokens when building span maps
- Extract and validate split patterns from pre-tokenizer configurations
- Add `bytes_char()` function implementing GPT-2 style byte-to-unicode conversion
- Use GPT2_PATTERN as default for bare ByteLevel pre-tokenizers
- Improve error messages with detailed context about unsupported configurations

```terminaloutput
crutcher@HeatLamp:~/git/wordchipper$ echo 'hello world' | cargo run -p wordchipper-cli -- cat --model 'cl100k_base' --encode
    Finished `dev` profile [unoptimized + debuginfo] target(s) in 0.19s
     Running `target/debug/wordchipper-cli cat --model cl100k_base --encode`
15339 1917 198
crutcher@HeatLamp:~/git/wordchipper$ echo 'hello world' | cargo run -p wordchipper-cli -- cat --model 'hf:Xenova/text-embedding-ada-002' --encode
    Finished `dev` profile [unoptimized + debuginfo] target(s) in 0.16s
     Running `target/debug/wordchipper-cli cat --model 'hf:Xenova/text-embedding-ada-002' --encode`
15339 1917 198
crutcher@HeatLamp:~/git/wordchipper$ echo 'hello world' | cargo run -p wordchipper-cli -- cat --model 'openai:gpt2' --encode
    Finished `dev` profile [unoptimized + debuginfo] target(s) in 0.16s
     Running `target/debug/wordchipper-cli cat --model 'openai:gpt2' --encode`
31373 995 198
crutcher@HeatLamp:~/git/wordchipper$ echo 'hello world' | cargo run -p wordchipper-cli -- cat --model 'hf:Xenova/gpt2' --encode
    Finished `dev` profile [unoptimized + debuginfo] target(s) in 0.16s
     Running `target/debug/wordchipper-cli cat --model 'hf:Xenova/gpt2' --encode`
31373 995 198
```
@crutcher
Make hf factory a better pass-through factory.
366dd4e 
Will now search hf for pretrained models if there is no schema.
@crutcher
Add VocabQuery::to_context() method and refactor HF vocab provider
7ed89a8 
- Add `VocabQuery::to_context()` to build cache context from schema, path, and name
- Make `VocabDescription::new()` generic over context and description string types
- Extract `vocab_from_hf_tokenizer()` as reusable conversion function
- Add special token handling to span_config instead of filtering them out
- Fix ByteLevel pre-tokenizer to require `use_regex=true` and use OA_GPT2_PATTERN
- Update HF vocab description context to use `to_context()` helper
- Add tests for `to_context()` with various query formats
@crutcher
Partial work on Qwen3.5.
b0e9475 
This is broken; the vocab seems to have aggregate tokens with no prior piecewise merges that are not special tokens. Which wordchipper enforces is never the case.

I need to investigate if this is true for this dataset; and explore what needs to change about wordchipper to handle this.

It may be that we need to support a situation where there multi-byte strings in the span map that are not in the byte map, but also *products* in the merge map (so they can only be produced by exact-match lookup, never by merges); but can still be members of merge pairs.

```terminaloutput
echo 'hello world' | RUST_BACKTRACE=1 cargo run -p wordchipper-cli -- cat --model Qwen/Qwen3.5-9B --encode
    Blocking waiting for file lock on artifact directory
   Compiling wordchipper v0.9.1 (/home/crutcher/git/wordchipper/crates/wordchipper)
   Compiling wordchipper-training v0.9.1 (/home/crutcher/git/wordchipper/crates/wordchipper-training)
   Compiling wordchipper-cli-util v0.9.1 (/home/crutcher/git/wordchipper/crates/wordchipper-cli-util)
   Compiling wordchipper-cli v0.9.1 (/home/crutcher/git/wordchipper/crates/wordchipper-cli)
    Finished `dev` profile [unoptimized + debuginfo] target(s) in 15.19s
     Running `target/debug/wordchipper-cli cat --model Qwen/Qwen3.5-9B --encode`
Debug: Some(("ĠаÑįÑĢо", 157513))
Debug: {
    248067: AddedToken {
        content: "</tool_response>",
        single_word: false,
        lstrip: false,
        rstrip: false,
        normalized: false,
        special: false,
    },
    248064: AddedToken {
        content: "<|repo_name|>",
        single_word: false,
        lstrip: false,
        rstrip: false,
        normalized: false,
        special: false,
    },
    248046: AddedToken {
        content: "<|im_end|>",
        single_word: false,
        lstrip: false,
        rstrip: false,
        normalized: false,
        special: true,
    },
    248059: AddedToken {
        content: "</tool_call>",
        single_word: false,
        lstrip: false,
        rstrip: false,
        normalized: false,
        special: false,
    },
    248054: AddedToken {
        content: "<|vision_end|>",
        single_word: false,
        lstrip: false,
        rstrip: false,
        normalized: false,
        special: true,
    },
    248044: AddedToken {
        content: "<|endoftext|>",
        single_word: false,
        lstrip: false,
        rstrip: false,
        normalized: false,
        special: true,
    },
    248058: AddedToken {
        content: "<tool_call>",
        single_word: false,
        lstrip: false,
        rstrip: false,
        normalized: false,
        special: false,
    },
    248049: AddedToken {
        content: "<|box_start|>",
        single_word: false,
        lstrip: false,
        rstrip: false,
        normalized: false,
        special: true,
    },
    248057: AddedToken {
        content: "<|video_pad|>",
        single_word: false,
        lstrip: false,
        rstrip: false,
        normalized: false,
        special: true,
    },
    248056: AddedToken {
        content: "<|image_pad|>",
        single_word: false,
        lstrip: false,
        rstrip: false,
        normalized: false,
        special: true,
    },
    248062: AddedToken {
        content: "<|fim_suffix|>",
        single_word: false,
        lstrip: false,
        rstrip: false,
        normalized: false,
        special: false,
    },
    248050: AddedToken {
        content: "<|box_end|>",
        single_word: false,
        lstrip: false,
        rstrip: false,
        normalized: false,
        special: true,
    },
    248055: AddedToken {
        content: "<|vision_pad|>",
        single_word: false,
        lstrip: false,
        rstrip: false,
        normalized: false,
        special: true,
    },
    248048: AddedToken {
        content: "<|object_ref_end|>",
        single_word: false,
        lstrip: false,
        rstrip: false,
        normalized: false,
        special: true,
    },
    248045: AddedToken {
        content: "<|im_start|>",
        single_word: false,
        lstrip: false,
        rstrip: false,
        normalized: false,
        special: true,
    },
    248066: AddedToken {
        content: "<tool_response>",
        single_word: false,
        lstrip: false,
        rstrip: false,
        normalized: false,
        special: false,
    },
    248068: AddedToken {
        content: "<think>",
        single_word: false,
        lstrip: false,
        rstrip: false,
        normalized: false,
        special: false,
    },
    248069: AddedToken {
        content: "</think>",
        single_word: false,
        lstrip: false,
        rstrip: false,
        normalized: false,
        special: false,
    },
    248052: AddedToken {
        content: "<|quad_end|>",
        single_word: false,
        lstrip: false,
        rstrip: false,
        normalized: false,
        special: true,
    },
    248053: AddedToken {
        content: "<|vision_start|>",
        single_word: false,
        lstrip: false,
        rstrip: false,
        normalized: false,
        special: true,
    },
    248060: AddedToken {
        content: "<|fim_prefix|>",
        single_word: false,
        lstrip: false,
        rstrip: false,
        normalized: false,
        special: false,
    },
    248047: AddedToken {
        content: "<|object_ref_start|>",
        single_word: false,
        lstrip: false,
        rstrip: false,
        normalized: false,
        special: true,
    },
    248051: AddedToken {
        content: "<|quad_start|>",
        single_word: false,
        lstrip: false,
        rstrip: false,
        normalized: false,
        special: true,
    },
    248061: AddedToken {
        content: "<|fim_middle|>",
        single_word: false,
        lstrip: false,
        rstrip: false,
        normalized: false,
        special: false,
    },
    248063: AddedToken {
        content: "<|fim_pad|>",
        single_word: false,
        lstrip: false,
        rstrip: false,
        normalized: false,
        special: false,
    },
    248065: AddedToken {
        content: "<|file_sep|>",
        single_word: false,
        lstrip: false,
        rstrip: false,
        normalized: false,
        special: false,
    },
}

thread 'main' (852780) panicked at crates/wordchipper/src/vocab/span_vocab.rs:247:51:
called `Result::unwrap()` on an `Err` value: VocabConflict("Pair (157513, 157853) -> 229601 parent 157513 is not defined")
stack backtrace:
   0: __rustc::rust_begin_unwind
             at /rustc/59807616e1fa2540724bfbac14d7976d7e4a3860/library/std/src/panicking.rs:689:5
   1: core::panicking::panic_fmt
             at /rustc/59807616e1fa2540724bfbac14d7976d7e4a3860/library/core/src/panicking.rs:80:14
   2: core::result::unwrap_failed
             at /rustc/59807616e1fa2540724bfbac14d7976d7e4a3860/library/core/src/result.rs:1867:5
   3: core::result::Result<T,E>::unwrap
             at /home/crutcher/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/core/src/result.rs:1233:23
   4: wordchipper::vocab::span_vocab::SpanMapVocab<T>::to_pair_vocab
             at ./crates/wordchipper/src/vocab/span_vocab.rs:247:51
   5: wordchipper::vocab::unified_vocab::UnifiedTokenVocab<T>::from_span_vocab
             at ./crates/wordchipper/src/vocab/unified_vocab.rs:85:37
   6: wordchipper::pretrained::huggingface::hf_factory::vocab_from_hf_tokenizer
             at ./crates/wordchipper/src/pretrained/huggingface/hf_factory.rs:199:18
   7: <wordchipper::pretrained::huggingface::hf_factory::HFVocabProvider as wordchipper::pretrained::factory::vocab_provider::VocabProvider>::load_vocab
             at ./crates/wordchipper/src/pretrained/huggingface/hf_factory.rs:245:29
   8: wordchipper::pretrained::factory::vocab_factory::VocabFactory::load_vocab
             at ./crates/wordchipper/src/pretrained/factory/vocab_factory.rs:229:28
   9: wordchipper::pretrained::factory::vocab_factory::load_vocab::{{closure}}
             at ./crates/wordchipper/src/pretrained/factory/vocab_factory.rs:99:55
  10: wordchipper::pretrained::factory::vocab_factory::with_vocab_factory
             at ./crates/wordchipper/src/pretrained/factory/vocab_factory.rs:76:5
  11: wordchipper::pretrained::factory::vocab_factory::load_vocab
             at ./crates/wordchipper/src/pretrained/factory/vocab_factory.rs:99:5
  12: wordchipper_cli_util::model_selector::ModelSelectorArgs::load_vocab
             at ./crates/wordchipper-cli-util/src/model_selector.rs:29:21
  13: wordchipper_cli_util::model_selector::ModelSelectorArgs::load_tokenizer
             at ./crates/wordchipper-cli-util/src/model_selector.rs:41:26
  14: wordchipper_cli::commands::cat_cmd::CatArgs::run
             at ./crates/wordchipper-cli/src/commands/cat_cmd.rs:51:45
  15: wordchipper_cli::Commands::run
             at ./crates/wordchipper-cli/src/main.rs:44:39
  16: wordchipper_cli::main
             at ./crates/wordchipper-cli/src/main.rs:18:18
  17: core::ops::function::FnOnce::call_once
             at /home/crutcher/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/core/src/ops/function.rs:250:5
note: Some details are omitted, run with `RUST_BACKTRACE=full` for a verbose backtrace.

```terminaloutput
$ echo 'hello world' | RUST_BACKTRACE=1 cargo run -p wordchipper-cli -- cat --model Qwen/Qwen3.5-9B --encode
    Blocking waiting for file lock on artifact directory
   Compiling wordchipper v0.9.1 (/home/crutcher/git/wordchipper/crates/wordchipper)
   Compiling wordchipper-training v0.9.1 (/home/crutcher/git/wordchipper/crates/wordchipper-training)
   Compiling wordchipper-cli-util v0.9.1 (/home/crutcher/git/wordchipper/crates/wordchipper-cli-util)
   Compiling wordchipper-cli v0.9.1 (/home/crutcher/git/wordchipper/crates/wordchipper-cli)
    Finished `dev` profile [unoptimized + debuginfo] target(s) in 15.19s
     Running `target/debug/wordchipper-cli cat --model Qwen/Qwen3.5-9B --encode`
Debug: Some(("ĠаÑįÑĢо", 157513))
Debug: {
    248067: AddedToken {
        content: "</tool_response>",
        single_word: false,
        lstrip: false,
        rstrip: false,
        normalized: false,
        special: false,
    },
    248064: AddedToken {
        content: "<|repo_name|>",
        single_word: false,
        lstrip: false,
        rstrip: false,
        normalized: false,
        special: false,
    },
    248046: AddedToken {
        content: "<|im_end|>",
        single_word: false,
        lstrip: false,
        rstrip: false,
        normalized: false,
        special: true,
    },
    248059: AddedToken {
        content: "</tool_call>",
        single_word: false,
        lstrip: false,
        rstrip: false,
        normalized: false,
        special: false,
    },
    248054: AddedToken {
        content: "<|vision_end|>",
        single_word: false,
        lstrip: false,
        rstrip: false,
        normalized: false,
        special: true,
    },
    248044: AddedToken {
        content: "<|endoftext|>",
        single_word: false,
        lstrip: false,
        rstrip: false,
        normalized: false,
        special: true,
    },
    248058: AddedToken {
        content: "<tool_call>",
        single_word: false,
        lstrip: false,
        rstrip: false,
        normalized: false,
        special: false,
    },
    248049: AddedToken {
        content: "<|box_start|>",
        single_word: false,
        lstrip: false,
        rstrip: false,
        normalized: false,
        special: true,
    },
    248057: AddedToken {
        content: "<|video_pad|>",
        single_word: false,
        lstrip: false,
        rstrip: false,
        normalized: false,
        special: true,
    },
    248056: AddedToken {
        content: "<|image_pad|>",
        single_word: false,
        lstrip: false,
        rstrip: false,
        normalized: false,
        special: true,
    },
    248062: AddedToken {
        content: "<|fim_suffix|>",
        single_word: false,
        lstrip: false,
        rstrip: false,
        normalized: false,
        special: false,
    },
    248050: AddedToken {
        content: "<|box_end|>",
        single_word: false,
        lstrip: false,
        rstrip: false,
        normalized: false,
        special: true,
    },
    248055: AddedToken {
        content: "<|vision_pad|>",
        single_word: false,
        lstrip: false,
        rstrip: false,
        normalized: false,
        special: true,
    },
    248048: AddedToken {
        content: "<|object_ref_end|>",
        single_word: false,
        lstrip: false,
        rstrip: false,
        normalized: false,
        special: true,
    },
    248045: AddedToken {
        content: "<|im_start|>",
        single_word: false,
        lstrip: false,
        rstrip: false,
        normalized: false,
        special: true,
    },
    248066: AddedToken {
        content: "<tool_response>",
        single_word: false,
        lstrip: false,
        rstrip: false,
        normalized: false,
        special: false,
    },
    248068: AddedToken {
        content: "<think>",
        single_word: false,
        lstrip: false,
        rstrip: false,
        normalized: false,
        special: false,
    },
    248069: AddedToken {
        content: "</think>",
        single_word: false,
        lstrip: false,
        rstrip: false,
        normalized: false,
        special: false,
    },
    248052: AddedToken {
        content: "<|quad_end|>",
        single_word: false,
        lstrip: false,
        rstrip: false,
        normalized: false,
        special: true,
    },
    248053: AddedToken {
        content: "<|vision_start|>",
        single_word: false,
        lstrip: false,
        rstrip: false,
        normalized: false,
        special: true,
    },
    248060: AddedToken {
        content: "<|fim_prefix|>",
        single_word: false,
        lstrip: false,
        rstrip: false,
        normalized: false,
        special: false,
    },
    248047: AddedToken {
        content: "<|object_ref_start|>",
        single_word: false,
        lstrip: false,
        rstrip: false,
        normalized: false,
        special: true,
    },
    248051: AddedToken {
        content: "<|quad_start|>",
        single_word: false,
        lstrip: false,
        rstrip: false,
        normalized: false,
        special: true,
    },
    248061: AddedToken {
        content: "<|fim_middle|>",
        single_word: false,
        lstrip: false,
        rstrip: false,
        normalized: false,
        special: false,
    },
    248063: AddedToken {
        content: "<|fim_pad|>",
        single_word: false,
        lstrip: false,
        rstrip: false,
        normalized: false,
        special: false,
    },
    248065: AddedToken {
        content: "<|file_sep|>",
        single_word: false,
        lstrip: false,
        rstrip: false,
        normalized: false,
        special: false,
    },
}

thread 'main' (852780) panicked at crates/wordchipper/src/vocab/span_vocab.rs:247:51:
called `Result::unwrap()` on an `Err` value: VocabConflict("Pair (157513, 157853) -> 229601 parent 157513 is not defined")
stack backtrace:
   0: __rustc::rust_begin_unwind
             at /rustc/59807616e1fa2540724bfbac14d7976d7e4a3860/library/std/src/panicking.rs:689:5
   1: core::panicking::panic_fmt
             at /rustc/59807616e1fa2540724bfbac14d7976d7e4a3860/library/core/src/panicking.rs:80:14
   2: core::result::unwrap_failed
             at /rustc/59807616e1fa2540724bfbac14d7976d7e4a3860/library/core/src/result.rs:1867:5
   3: core::result::Result<T,E>::unwrap
             at /home/crutcher/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/core/src/result.rs:1233:23
   4: wordchipper::vocab::span_vocab::SpanMapVocab<T>::to_pair_vocab
             at ./crates/wordchipper/src/vocab/span_vocab.rs:247:51
   5: wordchipper::vocab::unified_vocab::UnifiedTokenVocab<T>::from_span_vocab
             at ./crates/wordchipper/src/vocab/unified_vocab.rs:85:37
   6: wordchipper::pretrained::huggingface::hf_factory::vocab_from_hf_tokenizer
             at ./crates/wordchipper/src/pretrained/huggingface/hf_factory.rs:199:18
   7: <wordchipper::pretrained::huggingface::hf_factory::HFVocabProvider as wordchipper::pretrained::factory::vocab_provider::VocabProvider>::load_vocab
             at ./crates/wordchipper/src/pretrained/huggingface/hf_factory.rs:245:29
   8: wordchipper::pretrained::factory::vocab_factory::VocabFactory::load_vocab
             at ./crates/wordchipper/src/pretrained/factory/vocab_factory.rs:229:28
   9: wordchipper::pretrained::factory::vocab_factory::load_vocab::{{closure}}
             at ./crates/wordchipper/src/pretrained/factory/vocab_factory.rs:99:55
  10: wordchipper::pretrained::factory::vocab_factory::with_vocab_factory
             at ./crates/wordchipper/src/pretrained/factory/vocab_factory.rs:76:5
  11: wordchipper::pretrained::factory::vocab_factory::load_vocab
             at ./crates/wordchipper/src/pretrained/factory/vocab_factory.rs:99:5
  12: wordchipper_cli_util::model_selector::ModelSelectorArgs::load_vocab
             at ./crates/wordchipper-cli-util/src/model_selector.rs:29:21
  13: wordchipper_cli_util::model_selector::ModelSelectorArgs::load_tokenizer
             at ./crates/wordchipper-cli-util/src/model_selector.rs:41:26
  14: wordchipper_cli::commands::cat_cmd::CatArgs::run
             at ./crates/wordchipper-cli/src/commands/cat_cmd.rs:51:45
  15: wordchipper_cli::Commands::run
             at ./crates/wordchipper-cli/src/main.rs:44:39
  16: wordchipper_cli::main
             at ./crates/wordchipper-cli/src/main.rs:18:18
  17: core::ops::function::FnOnce::call_once
             at /home/crutcher/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/core/src/ops/function.rs:250:5
note: Some details are omitted, run with `RUST_BACKTRACE=full` for a verbose backtrace.
```
@crutcher
Update partial support for HF with basic errors.
460dacf
@crutcher
Fix no-std build.
acf7aef
@kylegrover
ground BPE pair vocab & add qwen specific logos lexer
024e948 
- build pair maps only from grounded BPE parents so pruned vocabularies stay valid
- add Qwen3.5 Hugging Face patterns and logos and regex-automata accelerators
- register regex-automata transforms through inventory hooks instead of a central table
@kylegrover
improve qwen lexer integration - register regex-automata transforms v…
17b7611 
…ia inventory hooks

add RegexAutomataTransformHook alongside existing lexer accelerator hooks
move OpenAI and Qwen regex-automata transform registration into their pattern modules
remove the HF-specific Qwen pattern canonicalization from the loader
@kylegrover
fix HF Qwen loading parity and add benchmark coverage
bbf7e62
@kylegrover
move HF input normalization ownership to UnifiedTokenVocab
2577054
@kylegrover kylegrover mentioned this pull request May 12, 2026
Open
@codecov

codecov Bot commented May 12, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 61.01695% with 207 lines in your changes missing coverage. Please review.

Files with missing lines Patch % Lines
...rdchipper/src/pretrained/huggingface/hf_factory.rs 15.46% 164 Missing ⚠️
...rdchipper/src/spanners/span_lexers/accelerators.rs 50.00% 11 Missing ⚠️
...hipper/src/pretrained/factory/vocab_description.rs 0.00% 9 Missing ⚠️
crates/wordchipper/src/vocab/unified_vocab.rs 87.27% 7 Missing ⚠️
crates/wordchipper/src/vocab/pair_vocab.rs 0.00% 6 Missing ⚠️
...rdchipper/src/spanners/span_lexers/logos/qwen35.rs 96.77% 4 Missing ⚠️
...chipper/src/spanners/span_lexers/regex_automata.rs 63.63% 4 Missing ⚠️
crates/wordchipper/src/support/normalization.rs 95.34% 2 Missing ⚠️

📢 Thoughts on this report? Let us know!

@kylegrover

kylegrover commented May 12, 2026

Copy link
Copy Markdown
Author

continuing some experiments on https://github.com/kylegrover/wordchipper/tree/kyleg/gemma4

@kylegrover

kylegrover commented May 13, 2026 •
edited
Loading

Copy link
Copy Markdown
Author

see also
kylegrover#1
kylegrover#2
(using my fork as a drafting space before PRing here)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@kylegrover @crutcher