New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

transformers-4.45版本以上Qwen2RotaryEmbedding.forward接口报错 #978

Open

lixiao2010 opened this issue Dec 26, 2024 · 0 comments

lixiao2010 commented Dec 26, 2024

xtuner/xtuner/model/modules/dispatch/qwen2.py

Line 266 in 90192ff

cos, sin = self.rotary_emb(value_states, seq_len=rotary_seq_len)

xtuner Qwen2的代码里面Qwen2RotaryEmbedding.forward接口接收seq_len参数

Qwen2RotaryEmbedding是从transformers里面import的，但是transformers从4.45版本以上该接口不接收seq_len参数

换成了position_ids，因此用xtuner训练模型会报错

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment