Your question
I want to ingest a checkpoint from HF into Megatron LM and then continue training on that. For the latter part (training) I will need TP > 1 or PP > 1 (Given the model size and the gpu memory i have). For this, when I convert the HF checkpoint to work with Megatron, I need the TP and PP values to match what I need in the training part.
However, right now the conversion scripts from HF to mcore seems to take in PP = 1 and TP = 1 (I am hoping I am mistaken here). How do I use the conversion scripts in tools/checkpoints/convert.py so I may be able to use TP > 1 and/or PP > 1?
Thanks.
Edit:
I am guessing this is answered (in the negative that no, there is no way currently to do this conversion) by #296 (comment) -- wondering if we have any updates here.
Your question
I want to ingest a checkpoint from HF into Megatron LM and then continue training on that. For the latter part (training) I will need TP > 1 or PP > 1 (Given the model size and the gpu memory i have). For this, when I convert the HF checkpoint to work with Megatron, I need the TP and PP values to match what I need in the training part.
However, right now the conversion scripts from HF to mcore seems to take in PP = 1 and TP = 1 (I am hoping I am mistaken here). How do I use the conversion scripts in
tools/checkpoints/convert.pyso I may be able to use TP > 1 and/or PP > 1?Thanks.
Edit:
I am guessing this is answered (in the negative that no, there is no way currently to do this conversion) by #296 (comment) -- wondering if we have any updates here.