[BUGFIX] Raise an error for no draft token case when draft_tp>1 #6369
Conversation
|
@cadedaniel @zifeitong @comaniac
|
wooyeonlee0
force-pushed
the
temporal-fix-skip-spec
branch
from
b9af049
to
ad8390c
Compare
wooyeonlee0
force-pushed
the
temporal-fix-skip-spec
branch
from
96ec39e
to
02dc475
Compare
|
@comaniac Thanks for the review :) But I'm not sure, the CI seems to have stopped. |
DarkLight1337
added
the
ready
|
@comaniac The CI test passed :) |
| if not self.allow_no_draft_tokens and sum( | ||
| proposals.proposal_lens) == 0: |
There was a problem hiding this comment.
I'm a bit worry about the overhead that sum brings, but I feel it's fine for now given that it won't be triggered with draft model TP=1. wdyt @cadedaniel?
There was a problem hiding this comment.
yeah, is it possible to store this field in proposals when we create proposals? that way we don't need an additional CPU-GPU-CPU sync
| """ | ||
| self.proposer_worker = proposer_worker | ||
| self.scorer_worker = scorer_worker | ||
| self.disable_by_batch_size = disable_by_batch_size or float("inf") | ||
| self.spec_decode_sampler = spec_decode_sampler | ||
| self.allow_no_draft_tokens = allow_zero_draft_token_step |
There was a problem hiding this comment.
nit: can we mark this private, e.g. _allow_no_draft_tokens? we should have done this for all properties but we missed it
| if not self.allow_no_draft_tokens and sum( | ||
| proposals.proposal_lens) == 0: |
There was a problem hiding this comment.
yeah, is it possible to store this field in proposals when we create proposals? that way we don't need an additional CPU-GPU-CPU sync
|
Thanks for the review! I'm gonna handle it right now :) |
wooyeonlee0
force-pushed
the
temporal-fix-skip-spec
branch
5 times, most recently
from
2861e99
to
3876858
Compare
|
@comaniac Is there any problem in CI now? |
wooyeonlee0
force-pushed
the
temporal-fix-skip-spec
branch
from
3876858
to
4b338d2
Compare
|
The failure you posted seems random. I'll monitor the current CI run and manually retry failed ones. |
|
@cadedaniel @comaniac I've updated the code as your suggestion and I think the code passed the test. @comaniac CI has finished. Would you retry the 'documentation-build' test? Thank you! |
|
@simon-mo can we get a force merge, doc build seems broken |
This PR adds a simple patch to raise an error to prevent users from hitting the hang error stated in #5814.
This error happens only when the skip speculation feature is activated and there are no generated draft tokens for "all" sequences in a step, and draft_tp > 1.
We may revisit this issue later because it's not resolved completely.