Why is it recommend to set load_in_8bit: true for LORA finetuning?
#1611
Unanswered
rudolpheric
asked this question in
Q&A
Replies: 0 comments
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
-
I startet to experment lora finetuning and since I have enough memory and still the model always gets worse through lora finetuning I am wondering why this is the case. I saw a warning in the logs that it is recommended to quantize to 8 bit. Why is this recommended? Shouldn't the model loose performance through quantisation?
Beta Was this translation helpful? Give feedback.
All reactions