Llama2 Unsupervised Finetuning

[gtx-cyber]

Hi I was trying to perform unsupervised fine-tuning with NVIDIA NeMo Framework on my custom dataset of 3B tokens on a pretrained Llama2, and I had downloaded Llama2 7b from HuggingFace and converted it to NeMo config file- Uning megatron_gpt_continue_training.py and megatron_llama_config.yaml.
After various parameters for -
restore_from_path: /workspace/mount/llama2-7b-hf/llama2-7b.nemo OR /workspace/mount/llama2-7b-hf/
resume_from_checkpoint: /workspace/mount/llama2-7b.nemo

The model trains from scratch. I want NeMo to use the pretrained weights.

Also a feature request being gradual unfreezing

[lhb8125]

Specify the .nemo ckpt for restore_from_path and ignore resume_from_checkpoint.

[AnirudhVIyer]

@gtx-cyber were you able tp resolve this?