Optimize 1st token for beam_search

[jiqing-feng]

We can repeat the outputs after the first token instead of repeating the first token inputs. The outputs are identical because the 1st token inputs are the same in each beam.

It could avoid extra computation.

I will update the rest of the generation methods once we agree on this change.

[amyeroberts]

cc @gante @zucchini-nlp

[jiqing-feng]

Hi @gante @zucchini-nlp . Could you please take a review? Thx!

[gante]

Hi @jiqing-feng 👋

The PR moves in the right direction, but we will want a different interface for it -- see "prefill" in this tracker. The prefill change is quite sensible with respect to interface, so we'll want to work on it ourselves. However, we'll be open to contributions after we add the prefill stage 🤗

[jiqing-feng]

Hi @jiqing-feng 👋

The PR moves in the right direction, but we will want a different interface for it -- see "prefill" in this tracker. The prefill change is quite sensible with respect to interface, so we'll want to work on it ourselves. However, we'll be open to contributions after we add the prefill stage 🤗

Thx! Please let me know when the PR is ready :)

[github-actions]

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the contributing guidelines are likely to be ignored.