Compute metrics while using SFT trainer

[shubhamagarwal92]

Hi @philschmid!

If I would like to compute metrics while training (similar to https://www.philschmid.de/fine-tune-flan-t5), how should I change the compute metrics function to pass to SFT trainer for Llamav2 (https://www.philschmid.de/instruction-tune-llama-2)

Would this be same as:

def compute_metrics(eval_pred, tokenizer):
    predictions, labels = eval_pred
    decoded_preds = tokenizer.batch_decode(predictions, skip_special_tokens=True)
    labels = np.where(labels != -100, labels, tokenizer.pad_token_id)
    decoded_labels = tokenizer.batch_decode(labels, skip_special_tokens=True)

    result = rouge.compute(predictions=decoded_preds, references=decoded_labels, use_stemmer=True)

    prediction_lens = [np.count_nonzero(pred != tokenizer.pad_token_id) for pred in predictions]
    result["gen_len"] = np.mean(prediction_lens)

    return {k: round(v, 4) for k, v in result.items()}

Specifically, do I still need to do

labels = np.where(labels != -100, labels, tokenizer.pad_token_id)

Is there a way to save the predictions decoded_preds while also passing the filename=f"{savedir}/{preds}.csv" to Trainer?

[philschmid]

You can write a callback for the Trainer which is executed after an evaluation phase. https://huggingface.co/docs/transformers/main_classes/callback#transformers.TrainerCallback.on_evaluate