This recipe steps you through how to finetune a Meta Llama 3 model on the text summarization task using the samsum dataset on a single GPU.
These are the instructions for using the canonical finetuning script in the llama-recipes package.
Ensure that you have installed the llama-recipes package (details).
To run fine-tuning on a single GPU, we will make use of two packages:
- PEFT to use parameter-efficient finetuning.
- bitsandbytes for int8 quantization.
python -m finetuning.py --use_peft --peft_method lora --quantization --use_fp16 --model_name /patht_of_model_folder/8B --output_dir Path/to/save/PEFT/model
The args used in the command above are:
--use_peft
boolean flag to enable PEFT methods in the script--peft_method
to specify the PEFT method, here we uselora
other options arellama_adapter
,prefix
.--quantization
boolean flag to enable int8 quantization
Note
In case you are using a multi-GPU machine please make sure to only make one of them visible using export CUDA_VISIBLE_DEVICES=GPU:id
.
Currently 3 open source datasets are supported that can be found in Datasets config file. You can also use your custom dataset (more info here).
-
grammar_dataset
: use this notebook to pull and process the Jfleg and C4 200M datasets for grammar checking. -
alpaca_dataset
: to get this open source data please download theaplaca.json
todataset
folder.
wget -P ../../src/llama_recipes/datasets https://raw.githubusercontent.com/tatsu-lab/stanford_alpaca/main/alpaca_data.json
samsum_dataset
to run with each of the datasets set the dataset
flag in the command as shown below:
# grammer_dataset
python -m finetuning.py --use_peft --peft_method lora --quantization --dataset grammar_dataset --model_name /patht_of_model_folder/8B --output_dir Path/to/save/PEFT/model
# alpaca_dataset
python -m finetuning.py --use_peft --peft_method lora --quantization --dataset alpaca_dataset --model_name /patht_of_model_folder/8B --output_dir Path/to/save/PEFT/model
# samsum_dataset
python -m finetuning.py --use_peft --peft_method lora --quantization --dataset samsum_dataset --model_name /patht_of_model_folder/8B --output_dir Path/to/save/PEFT/model
To help with benchmarking effort, we are adding the support for counting the FLOPS during the fine-tuning process. You can achieve this by setting --flop_counter
when launching your single/multi GPU fine-tuning. Use --flop_counter_start
to choose which step to count the FLOPS. It is recommended to allow a warm-up stage before using the FLOPS counter.
Similarly, you can set --use_profiler
flag and pass a profiling output path using --profiler_dir
to capture the profile traces of your model using PyTorch profiler. To get accurate profiling result, the pytorch profiler requires a warm-up stage and the current config is wait=1, warmup=2, active=3, thus the profiler will start the profiling after step 3 and will record the next 3 steps. Therefore, in order to use pytorch profiler, the --max-train-step has been greater than 6. The pytorch profiler would be helpful for debugging purposes. However, the --flop_counter
and --use_profiler
can not be used in the same time to ensure the measurement accuracy.