diff --git a/docs/src/python/tokenizing.rst b/docs/src/python/tokenizing.rst index a82b8a6..e69e956 100644 --- a/docs/src/python/tokenizing.rst +++ b/docs/src/python/tokenizing.rst @@ -52,3 +52,6 @@ or a plain text vocabulary file. core.CharTrie tokenizer_helpers.read_trie_from_vocab tokenizer_helpers.read_trie_from_spm + tokenizer_helpers.read_bpe_from_spm + tokenizer_helpers.read_bpe_from_hf + tokenizer_helpers.gpt2_byte_map