Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Will Embeding change? #122

Open
Z-Zili opened this issue Dec 14, 2023 · 6 comments
Open

Will Embeding change? #122

Z-Zili opened this issue Dec 14, 2023 · 6 comments

Comments

@Z-Zili
Copy link

Z-Zili commented Dec 14, 2023

I have been studying LoRA recently and I noticed that during pre-training, the word vectors change as the training progresses. However, what about when using LoRA for fine-tuning? Do the word vectors still change, or is it only the attention weights.

@yezhengmao1
Copy link
Collaborator

Typically, fine-tuning with LoRA does not alter the embedding layer. However, in some models, the training process of the LoRA module may concurrently train the embedding layer (due to the need to add additional Chinese tokens).

@Z-Zili
Copy link
Author

Z-Zili commented Dec 14, 2023

Typically, fine-tuning with LoRA does not alter the embedding layer. However, in some models, the training process of the LoRA module may concurrently train the embedding layer (due to the need to add additional Chinese tokens).

So, what you mean is that when fine-tuning, if there is a word in the dataset that the model has not seen before, we need to adjust the embedding layer?

@yezhengmao1
Copy link
Collaborator

Typically, fine-tuning with LoRA does not alter the embedding layer. However, in some models, the training process of the LoRA module may concurrently train the embedding layer (due to the need to add additional Chinese tokens).

So, what you mean is that when fine-tuning, if there is a word in the dataset that the model has not seen before, we need to adjust the embedding layer?

yes, the best way is to add the new token ids to the tokenizer model and then enable the embedding layer's grad to train the LoRA module.

@Z-Zili
Copy link
Author

Z-Zili commented Dec 15, 2023

OK,Thank you for your answer

@owos
Copy link

owos commented Sep 9, 2024

THank you for the response @yezhengmao1 . What if I want to use lora to update the embedding layers?

@yezhengmao1
Copy link
Collaborator

THank you for the response @yezhengmao1 . What if I want to use lora to update the embedding layers?

the embedding is just lookup table, can not use the lora to it.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

3 participants