README.md

🧑‍🏫🤏 LoRA-Instruct

This repository contains code for fine-tuning permissive open source LLMs using low-rank adaptation (LoRA).

Code is tested using Stanford Alpaca dataset.

• Estimated training time for fine-tuning RedPajama-INCITE-Base-7B-v0.1 with a single RTX 3090 and Stanford Alpaca is ~12 hours.
• Estimated training time for fine-tuning RedPajama-INCITE-Base-7B-v0.1 with RTX 3090 and RTX Titan and Stanford Alpaca is ~6.5 hours.
• Currently only supports LoRA Instruct fine-tuning RedPajama-INCITE-Base-7B-v0.1.

Inspired by Alpaca-LoRA

Trained Models

Model	Runs	Training Time	Link
LLaMA 3B	⬜
LLaMA 7B	⬜
RedPajama 3B	✅	1:44:14
RedPajama 7B	✅	3:09:58
MPT 3B	⬜
MPT 7B	⬜
Falcon 7B	✅

Training Hardware Spec

Ubuntu 20.04.1 LTS (WSL2)

Driver Version: 531.41
CUDA Version: 12.1
cuDNN version: 8.5.0

Local Setup

Install dependencies

poetry install

To fine-tune using NVidia 2000 series GPU or earlier, please comment out this line in finetune.py

model = prepare_model_for_int8_training(model)

Training (finetune.py)

This file contains a straightforward application of PEFT / LoRA to decoder only model, as well as some code related to prompt construction and tokenization.

Example usage:

python finetune.py \
    --base_model 'togethercomputer/RedPajama-INCITE-Base-7B-v0.1' \
    --output_dir './lora-redpajama'

Distributed Training with 🤗 Accelerate

We uses HuggingFace's accelerate library for distributed training. The following is an example for distributed training with two GPUs.

• NOTE: please set the following environment variables

export WORLD_SIZE=2
export CUDA_VISIBLE_DEVICES=0,1

torchrun \
    --nproc_per_node=2 \
    --master_port=1234 \
    finetune.py \
    --base_model 'togethercomputer/RedPajama-INCITE-Base-7B-v0.1' \
    --output_dir './lora-redpajama'

References

• LoRA: Low-Rank Adapation of Large Language Models
• State-of-the-art Parameter-Efficient Fine-Tuning (PEFT) methods
• Stanford Alpaca: An Instruction-following LLaMA Model
• EMNLP 2022 Tutorial: Modular and Parameter-Efficient Fine-Tuning for NLP Models