Do we need a config to change padding_side='left
before the evaluation?
#31672
Labels
Feature request
Request for a new feature
Feature request
I am trying to train a Llama model (a decoder-only model). I want to evaluate my model with not only the loss but also some generation-based metric. For example, my eval dataset could be a str as
1+2=
, and I use the Seq2seqTrainer which provides the modified prediction step so I can get the prediction of the model in theEvalPrediction
. Then I write my eval code in the functioncompute_metrics
and provide it for the Seq2seqTrainer.The problem is about the padding_side of the tokenizer. Because I need to train the model, the tokenizer should be right padding in training dataset. (Because it is the default setting of Llama.) However, when I try to evaluate the model, the tokenizer should be changed into left padding because I need my model to generate. I do not find a easy way to do this, unless I change the source code of the trainer (for example, the
get_eval_dataloader
method of the Trainer).My questions are:
Motivation
Motivation: generation-based evaluation when we train a decoder-only autoregressive model like llama.
Your contribution
I do not know what I can help.
The text was updated successfully, but these errors were encountered: