Differnt Tokenizer

[dustinwloring1988]

Your question
Is there a way to start training on a llama2 with a llama3 tokenizer? I plan on doing all the pretraining myself, if so and someone can provide any information please let me know.

[Advait251206]

Yes, it's possible, but it's not as simple as swapping the tokenizer. The tokenizer defines the model's vocabulary, so changing it also changes the embedding matrices.

When moving from a Llama 2 tokenizer to a Llama 3 tokenizer, you need to consider the following:

1. Vocabulary size changes

   • Llama 3 uses a different tokenizer and vocabulary than Llama 2.
   • As a result, the input embedding layer (embed_tokens) and the output language modeling head (lm_head, if not tied) will have different shapes.

2. Loading a Llama 2 checkpoint

   • You can reuse most of the transformer weights (attention layers, MLPs, layer norms, etc.).
   • However, the embedding weights (and output head if untied) cannot be loaded directly because their dimensions no longer match.
   • Typically, you would:
     • Load all compatible weights.
     • Randomly initialize the new embedding/output layers.
     • Continue pretraining so the new token embeddings are learned.

3. Tokenizer differences

   • The Llama 3 tokenizer doesn't just have a larger vocabulary—it uses different tokenization rules and merges.
   • Existing embeddings from the Llama 2 tokenizer don't have a one-to-one correspondence with Llama 3 tokens, so there isn't a straightforward mapping between the two vocabularies.

4. Megatron-LM support

   • Megatron-LM itself doesn't prevent you from using a different tokenizer, provided your tokenizer implementation is supported (or you add support for it).
   • The main work is ensuring the model configuration (vocab_size) matches the tokenizer and handling the embedding initialization appropriately when loading a checkpoint.

If you're pretraining from scratch, then this is much simpler:

• Use the Llama 3 tokenizer from the beginning.
• Set vocab_size to match the tokenizer.
• Initialize all model weights normally.
• Train from scratch.

If your goal is to continue pretraining from an existing Llama 2 checkpoint, expect to reuse the transformer blocks while reinitializing the token embedding layer (and output head if necessary). The model will then need additional training to learn representations for the new vocabulary.

---

If this answer helped or pointed you in the right direction, I'd appreciate it if you could mark it as the accepted answer so it's easier for others with the same issue to find.

Also, if you found my contribution useful, I'd appreciate it if you could check out my GitHub profile, follow me, and star any repositories you find interesting.

GitHub: https://github.com/Advait251206