Your question
If I change the seq-length during training, like 4096 -> 8192, or 4096 -> 1024, will the data loader reset the order of the dataset or continue with the old order?
I want to change the seq-length after training maybe 100k steps, and I need the model to continue train with the left data, will Megatron-LM do this by default?
Your question
If I change the
seq-lengthduring training, like 4096 -> 8192, or 4096 -> 1024, will the data loader reset the order of the dataset or continue with the old order?I want to change the
seq-lengthafter training maybe 100k steps, and I need the model to continue train with the left data, will Megatron-LM do this by default?