create test.py#364
Conversation
Goal: 1.Saving our time to test the environment. 2.Demonstrating how to fine-tune small dataset. Features: There 10 instances in the file that can be fine-tuned in 1:30s with RTX3090 and 4mins in Colab T4 16G , the outcome is 100% correct and I didn't find any overfitting for Llama-7B yet. Command: python test.py (That's all, the requirement packages, dataset and generate function are all in this file)
Add some comment.
Referencing PR tloen#364 in original Repo
1. Add utils and template into file, we can paste or upload this single file to other cloud server for testing. 2. Add callback function to observe the text generation while evaluation, we randomly sample 3 instances from val_datas to observe the generation improvement with model update. 3.Changed peft to old version, it doesn't work again.
|
I ran it and got results, some examples of which are below. What could be the issue? I already installed the PEFT and Transformer versions that were mentioned. Sample 2: Predict: Sample 3: Predict: # offer description |
The results shown inefficient training, the model neither remember the answer nor template and eos_token_id. How's the other questions, can the model predict at least one correct answer out of the ten? |
|
These answers seems like you only ran 10 steps, did you train 70 steps? And the last prints are not the final model we generated, you need to test it in Gradio. I just ran the code again, here is the link:https://colab.research.google.com/drive/1_mhkjwfo8kafK5v97-chz4ySzZLNHfUH?usp=sharing |
|
You're right, thank you! |
|
Note: to be able to use this test script (or alpaca-lora training really) in free version of Google Colab, I had to split the yahma/llama-7b-hf into more shards because of RAM limitations. I uploaded the split version as jploski/llama-7b-hf. If I feed the final LoRA weights to generate.py, the outputs in Gradio are almost as expected ("What is your model?" produces "My pre-trained model is Llama-1B, finetuned via LORA." and "Are you overfitting?" produces "Of course nah", but others match exactly. |
|
I should add that this is a very worthwhile PR and something like this should be referred to from front page documentation. I wasted a LOT of time running into various problems because of limited hardware resources in Colab (which I at first did not realize they were too limited!) and decapoda-research/llama-7b-hf being no longer compatible. I think for beginners it's great to have something which tells you that your setup is correct. The problem is that as a beginner you don't even really know which outputs to expect and with which parameters to run, and you would like to start with a small example which does not require a big dataset, renting hardware and hours of training (maybe to get nothing in the end). It's easy to believe that it is yourself doing something wrong or fundamentally misunderstanding something rather than just bad configuration. So from educational viewpoint this self-contained minimal example script is indeed very interesting. |
3 participants
Goal:
Features:
There 10 instances in the file that can be fine-tuned in 1:30s with RTX3090 and 4mins in Colab T4 16G , the outcome is 100% correct and I didn't find any overfitting for Llama-7B yet.
Command:
python test.py
(That's all, the requirement packages, dataset, generate function, and other components like utils and template are all in this file, we can paste or upload this single file to any cloud server then run.)