Llama3.1 model requires transformers==4.31.0 and current NxD llama3 test case is not compatible with the version.

[KeitaW]

Llama example in NxD installs transformers==4.31.0 (ref) but that version of transformers cannot load Meta-Llama-3.1-70B.

from transformers import AutoTokenizer, AutoModelForCausalLM
model = AutoModelForCausalLM.from_pretrained("/fsx/ubuntu/Meta-Llama-3.1-70B")

causes

  File "/fsx/ubuntu/aws_neuron_venv_pytorch/lib/python3.8/site-packages/transformers/models/auto/configuration_auto.py", line 999, in from_pretrained
    return config_class.from_dict(config_dict, **unused_kwargs)
  File "/fsx/ubuntu/aws_neuron_venv_pytorch/lib/python3.8/site-packages/transformers/configuration_utils.py", line 744, in from_dict
    config = cls(**config_dict)
  File "/fsx/ubuntu/aws_neuron_venv_pytorch/lib/python3.8/site-packages/transformers/models/llama/configuration_llama.py", line 145, in __init__
    self._rope_scaling_validation()
  File "/fsx/ubuntu/aws_neuron_venv_pytorch/lib/python3.8/site-packages/transformers/models/llama/configuration_llama.py", line 163, in _rope_scaling_validation
    raise ValueError(
ValueError: `rope_scaling` must be a dictionary with with two fields, `name` and `factor`, got {'factor': 8.0, 'low_freq_factor': 1.0, 'high_freq_factor': 4.0, 'original_max_position_embeddings': 8192, 'rope_type': 'llama3'}

I had to install transformers==4.43.1 to resolve the issue. Unfortunately, this version of transformers does not have _init_rope in modeling_llama_nxd.py so the existing script may not work out of the box?

neuronx-distributed/examples/training/llama/modeling_llama_nxd.py

Line 280 in 977d3b7

self._init_rope()

v4.4.31.0: https://github.com/huggingface/transformers/blob/e42587f596181396e1c4b63660abf0c736b10dae/src/transformers/models/llama/modeling_llama.py#L258C1-L273C78

v4.43.1: https://github.com/huggingface/transformers/blob/782bfffb2e4dfb5bbe7940429215d794f4434172/src/transformers/models/llama/modeling_llama.py#L306

Steps used to retrieve model weights

Install huggingface_hub:

pip install huggingface_hub

this installs following huggingface_hub:

huggingface_hub version: 0.27.1

then use the followincg command to download model weights:

huggingface-cli download meta-llama/Meta-Llama-3.1-70B --local-dir /fsx/ubuntu/Meta-Llama-3.1-70B

[aws-bowencc]

Hi @KeitaW, thx for raising this issue due to transformers version update. We are looking into it and will get back to you.