[QUESTION] How to load checkpoint saved in one parallel configuration (tensor/pipeline/data parallelism) can be loaded in a different parallel configuration ?

[polisettyvarma]

How to load checkpoint saved in one parallel configuration (tensor/pipeline/data parallelism) can be loaded in a different parallel configuration ?

Based on this doc - https://github.com/NVIDIA/Megatron-LM/blob/main/docs/source/api-guide/dist_checkpointing.rst
There are some conflicting statements.

1. zarr is not default format as mentioned.
2. i tried 8 card config, src - tp2pp2dp2, dst - tp2pp1dp4 didn't work

Can you provide a working example end to end to showcase this feature, Thanks.

[wplf]

You can checkout convert in tools folder,tools/checkpoint/convert.py.
This may help you.

[polisettyvarma]

@wplf Thanks for the comment, can you please elaborate more on this ?

[polisettyvarma]

Can someone answer this ?