[PoC][Spec Decoding] Stop speculating when confidence is low #5914
Conversation
Just an experiment, this gives an additional ~3% speedup in a greedy bs=1 example. This just "pads" out proposed tokens up to the num_predict_tokens with the final predicted token. Would be good if we could change the framework to allow the proposer model to return a variable number of tokens. I'm not sure how this will work in non-greedy case - we'd still probably want to evaluate the cutoff based on a greedily sampled token.
njhill
force-pushed
the
spec-token-cutoff
branch
from
0f379df
to
789a608
Compare
|
I am a little hesitant to merge bs=1 optimizations like this until we have a picture of how QPS vs ITL looks. In particular this one will trade off QPS to achieve higher ITL reduction (a fine tradeoff if the user wants it). Can we put it behind a flag? |
|
@cadedaniel yes this is just exploration right now, have no intention of merging until further testing/hardening and understanding the implications. I'm not sure I follow the QPS vs ITL tradeoff point though, I'll give it some thought. I think all this optimization does really is to increase the acceptance rate (per proposed token). It would be less effective for bs > 1 because I'm only aborting when the probs for all sequences are below the cutoff value (will happen much less frequently), but it shouldn't be detrimental in theory. |
Just a quick experiment, this gives an additional ~3% speedup in a greedy bs=1 example.
This just "pads" out proposed tokens up to the num_predict_tokens with the final predicted token. Would be good if we could change the framework to allow the proposer model to return a variable number of tokens.
I'm not sure how this will work in non-greedy case - we'd still probably want to evaluate the cutoff based on a greedily sampled token.
@cadedaniel @LiuXiaoxuanPKU I guess this is related to DSD