IndexTTS2 tokenizer needs CJK pre-tokenization for Mandarin input

[jc119-chao]

Summary

The current IndexTTS2Tokenizer normalization handles Latin text, but continuous Mandarin input can remain in the form ▁你好.... With the published aufklarer/IndexTTS2-MLX-fp16 bundle, this can leave no valid SentencePiece path and produce IndexTTS2TokenizerError.unencodableText.

I encountered this while integrating the existing speech-swift IndexTTS2 runtime into a desktop dubbing pipeline. Applying CJK character pre-tokenization allowed real Mandarin synthesis to complete successfully.

Environment

• speech-swift: 7984666dd7dc9233132c57a09bd9bf490a2ae448
• model: aufklarer/IndexTTS2-MLX-fp16
• model revision: 208b3d6ea53a119f3501b3bfd8e666b9b5e8c705
• Apple M2, 24 GB RAM
• macOS 26.5
• Swift 6.3.2

Current behavior

normalizedPieceText(for:) currently:

1. applies compatibility normalization;
2. collapses whitespace;
3. uppercases the full string;
4. inserts ▁ only at whitespace boundaries.

For Mandarin without spaces, multiple CJK scalars therefore remain one continuous segment. This does not mirror the upstream IndexTTS2 text front end, which pre-tokenizes CJK characters before BPE lookup.

Example input:

你好，这是一个中文合成测试。

The model-compatible pre-tokenized shape is approximately:

▁你▁好▁,▁这▁是▁一▁个▁中▁文▁合▁成▁测▁试▁.

Local fix evaluated

The local integration currently applies:

• per-scalar CJK pre-tokenization;
• IndexTTS2-compatible punctuation normalization;
• locale-aware number expansion;
• an unknown-token fallback so one unsupported scalar does not invalidate the entire path.

This was enough to run real Mandarin synthesis through the Swift/MLX runtime and a persistent JSONL bridge. The surrounding desktop integration has 22 runtime checks, but the tokenizer change itself still needs focused speech-swift regression tests before a PR.

Proposed upstream scope

If this approach matches the intended text front end, I can prepare a focused PR containing only:

1. the tokenizer normalization fix;
2. regression tests for Mandarin, mixed Chinese/English, punctuation, numbers, and unknown scalars;
3. no application-specific bridge or dependency pinning.

Would you prefer the PR to mirror the upstream Python normalizer closely, or keep the Swift implementation limited to the minimum CJK boundary fix?

[ivan-digital]

Thanks for the report — fixed in #465, with #467 and #468 rounding it out, all on main.

Two findings worth sharing, since the root cause was a little different from the description:

• The unencodableText error came from punctuation, not from CJK continuity. bpe.model has all 8,467 CJK characters as bare single pieces, so 你好 already encoded; what had no piece was 。 (and ", (, :, ; — so plain English hit it too). The fix applies upstream's char_rep_map table before tokenization.
• The bigger problem was silent: upstream's tokenize_by_CJK_char puts a space before every CJK character, so the model expects ▁ 你 ▁ 好 ▁ 世 — a standalone ▁ before each character; there are no ▁你-style pieces in the vocab. The Swift path was producing ▁ 你 好 世. That's now mirrored exactly: 241 generated inputs match upstream sentencepiece token-for-token, and a Mandarin synthesis + Qwen3-ASR round trip comes back character-exact.

Unsupported scalars now degrade to a merged <unk> (as SentencePiece does) instead of failing the whole input. Numbers are read out in words before tokenization (#467 — the vocab has no digit pieces; upstream relies on WeTextProcessing for this), and long text is synthesized in punctuation-aligned segments like upstream (#468), which matters more now that per-character ▁ doubles Mandarin token counts.