How to start training again from last checkpoint

[FatimaArshad-DS]

Hi,

My training got interrupted and Im trying to restart training from last checkpoint. However, training starts from the beginning. How do I make it start from last checkpoint?

[titu1994]

If you're using exp manager, it has two flags for resuming training v

Without it, you can try loading your checkpoint using load_from_checkpoinr() and then calling trainer.train() again but we haven't tested that

[yunbin]

What is involved to restart training using speech_to_text_finetune.py for model type EncDecHybridRNNTCTCBPEModel? The script states: "Currently for simplicity of single script for all model types, we only support init_from_nemo_model and init_from_pretrained_model", but resume from a ckpt file is essential for any model building tool.

[FatimaArshad-DS]

Are you talking about this one? create_checkpoint_callback. It is set to True @titu1994

[itzsimpl]

@FatimaArshad-DS see
https://github.com/NVIDIA/NeMo/blob/8a172df69250da231742893387e9a1dd01b16e3d/nemo/utils/exp_manager.py#L197-L200
and
https://github.com/NVIDIA/NeMo/blob/8a172df69250da231742893387e9a1dd01b16e3d/nemo/utils/exp_manager.py#L204-L206

[itzsimpl]

Note that you will need to fix eithr
https://github.com/NVIDIA/NeMo/blob/8a172df69250da231742893387e9a1dd01b16e3d/nemo/utils/exp_manager.py#L188-L189
or set
https://github.com/NVIDIA/NeMo/blob/8a172df69250da231742893387e9a1dd01b16e3d/nemo/utils/exp_manager.py#L194-L195
to point to your specific run, that you wish to resume.

[ericharper]

We also some documentation here: https://docs.nvidia.com/deeplearning/nemo/user-guide/docs/en/stable/core/exp_manager.html

[XuesongYang]

You could continue training from the latest checkpoint following the steps:

1. go to your checkpoint directory, and rename the latest a.ckpt file to a-last.ckpt because https://github.com/NVIDIA/NeMo/blob/main/nemo/utils/exp_manager.py#L420 ;
2. override the default YAML config with ++exp_manager.resume_if_exists=True and run the same training recipe again.

[FatimaArshad-DS]

This worked!!!!
However, what happens if training is interrupted before the epoch ends? What that cause any issue? As far as I have observed it trains on already trained data

[XuesongYang]

if training is interrupted within an epoch, then no new checkpoint (a+1)-last.ckpt would be saved. So you have to load the previous a-last.ckpt again and train.

[FatimaArshad-DS]

Training on a-last.ckpt starts training on already seen data samples. How do I avoid it?

[XuesongYang]

it did not happen to me. Maybe change another random seed? @titu1994 are you aware of this observation?

[titu1994]

If you have not turned on shuffle, or have not used shuffle_n in train_ds, then data loader will not shuffle the data (if using tarred dataset).

[XuesongYang]

@FatimaArshad-DS Do you have any other questions about this?

[DevHector-Bit]

Its possible save checkpoint from Moe models? i got some issues trying to save one the only way i se now its saving weights but doesnt work for resume training