Use query in linear flags - seq as fallback option #396
Conversation
Signed-off-by: Agata Dobrzyniewicz <[email protected]>
adobrzyn
requested review from
afierka-intel,
kzawora-intel,
mgawarkiewicz-intel,
michalkuligowski,
mswiniarsk,
vivekgoe and
xuechendi
as code owners
🚧 CI BlockedThe main CI workflow was not started for the following reason:
|
Signed-off-by: Agata Dobrzyniewicz <[email protected]>
| for p, e, d in zip(params, env_vars, default_values): | ||
| val = os.environ.get(e) | ||
|
|
||
| if val is None and dim == 'query': |
There was a problem hiding this comment.
As far as I know we don't have VLLM_DECODE_SEQ_BUCKET_{p} - is there a need for making this code handle such a case ? This code looks like it would need to
There was a problem hiding this comment.
No, we don't set this dim - nor query nor seq, for decode
There was a problem hiding this comment.
Ok, so shouldn't we put also "and phase == 'prompt' and then set
fallback_env = f'VLLM_PROMPT_SEQ_BUCKET_{p}'.upper() in 102 line ?
Signed-off-by: Agata Dobrzyniewicz <[email protected]>
🚧 CI BlockedThe main CI workflow was not started for the following reason:
|
4 participants
No flag -> default
Query -> query value
Seq -> seq value + "will be depricated"warning
Query & seq -> query value