Huggingface already has an efficient implementation of this?

[laurislopata]

When Karpathy claimed an efficient implementation of the BPE optimizer doesn't exist, I did some research and found this on Hugging Face: https://github.com/huggingface/tokenizers/blob/main/tokenizers/src/models/bpe/trainer.rs

Isn't this exactly what Karpathy was creating?

[NLPV2011]

Hugging face's tokenizers not support for all non-english language

[AugustasMacijauskas]

Hugging face's tokenizers not support for all non-english language

I too am convinced that HF already supports training a BPE tokenizer, but I am relatively new to this, could you elaborate? I thought that any text can be put into their tokenizers and it just works?

[NLPV2011]

I too am convinced that HF already supports training a BPE tokenizer, but I am relatively new to this, could you elaborate? I thought that any text can be put into their tokenizers and it just works?

i'm not sure, i think i need to implement BPE tokenizer from scratch for easy to use.... you may like karpathy's minBPE