Is it possible to set repetition_penalty?

[ein-ich]

I guess there are 2 cases:

1. Locally to the gen
2. Considering the whole context

2 is more important for me (and it should be simple to implement?). It seems, however, like neither is possible?

I really like the library, but I'm using base LLaMa and not being able to set repetition_penalty makes it almost useless for me.

Am I missing something?

[slundberg]

Good question! I just added a new kwargs passthrough to the gen command to address this. Now you can pass anything through the transformers generate, like repetition_penalty. Just note that some parameters that change the output of the generate command would break things I think (like num_beams).

You should be able to use anything in the "Parameters for manipulation of the model output logits" section at https://huggingface.co/docs/transformers/main_classes/text_generation#transformers.GenerationConfig

This is in main now and will be part of 0.0.62 when it comes out.

[chrisfentiman]

Is there documentation on this? It was only by chance that I stumbled across this thread. If not, where would be the best place to add this? I can open PR.

[jelling]

Cheers for the new feature and +1 for docs. Take credit! :)

Also, for anyone else stumbling into this, the change is in main but has not yet been released via pypi. So to get the new code, you'll need to point pip to repo like this:

guidance @ git+https://github.com/microsoft/guidance.git

[AtharvanDogra]

Hi @slundberg this functionality seems to have vanished from the library. gun currently does not accept repetition or length penalty.