[ENHANCEMENT] Do you have a plan that supports Mixtral 8x7B? #1106

matrixssy started this conversation in Ideas

matrixssy
Jan 4, 2024

Replies: 7 comments

cdj0311
Jan 14, 2024

The same question.

0 replies

matrixssy
Jan 15, 2024
Author

The same question.

I am working for it, but I am not sure if it will be accepted.

0 replies

bityigoss
Jan 17, 2024

@matrixssy any progress ? thank you.

0 replies

matrixssy
Jan 18, 2024
Author

see #667

0 replies

yanring
Feb 15, 2024

Hi, Please refer to this script for MoE/Mixtral training.

0 replies

matrixssy
Feb 17, 2024
Author

Hi, Please refer to this script for MoE/Mixtral training.

Great! But I would like to know how to convert Hugging Face (HF) weights to Megatron (MG) format, and if it's possible to convert them back after training?

0 replies

github-actions[bot]
Bot Apr 17, 2024

Marking as stale. No activity in 60 days.

0 replies

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment