Update of VLLM_PROMPT_BS_BUCKET_MAX logic, real bs change, not only linear warmup #348
Conversation
…inear warmup Signed-off-by: Iryna Boiko <[email protected]>
iboiko-habana
requested review from
adobrzyn,
afierka-intel,
kzawora-intel,
mgawarkiewicz-intel,
michalkuligowski,
mswiniarsk,
vivekgoe and
xuechendi
as code owners
| logger().warning("*******************************************************") | ||
| for flag in user_flags: | ||
| if flag in ("VLLM_PROMPT_BS_BUCKET_MAX"): | ||
| continue |
There was a problem hiding this comment.
If I add this flag only wouldn't I have those stars **** with no warning?
There was a problem hiding this comment.
Thanks. Updated
docs/configuration/env_vars.md
Outdated
| - `VLLM_GRAPH_DECODE_STRATEGY`: strategy determining order of decode graph capture, `min_tokens` or `max_bs`. The default is `max_bs`. | ||
| - `VLLM_EXPONENTIAL_BUCKETING`: if `true`, enables exponential bucket spacing instead of linear. The default is `true`. | ||
| - `VLLM_PROMPT_BS_BUCKET_MAX`: prefill batch size max. The default is `1`. | ||
| - `VLLM_{phase}_{dim}_BUCKET_{param}`: collection of 12 environment variables configuring ranges of bucketing mechanism (linear bucketing only). |
There was a problem hiding this comment.
We no longer have here 12 environment variables (even before your change there were more). Let's either don't mention a number or fix it to a proper value
There was a problem hiding this comment.
Thanks. Updated
docs/configuration/env_vars.md
Outdated
| - `VLLM_PROMPT_BS_BUCKET_MAX`: prefill batch size max. The default is `1`. | ||
| - `VLLM_{phase}_{dim}_BUCKET_{param}`: collection of 12 environment variables configuring ranges of bucketing mechanism (linear bucketing only). | ||
| - `{phase}` is either `PROMPT` or `DECODE` | ||
| - `{dim}` is either `BS`, `SEQ` or `BLOCK` |
There was a problem hiding this comment.
according to the env variables we list here dim can also be CTX
There was a problem hiding this comment.
Thanks. Updated
docs/configuration/env_vars.md
Outdated
| - `VLLM_GRAPH_PROMPT_STRATEGY`: strategy determining order of prompt graph capture, `min_tokens` or `max_bs`. The default is `min_tokens`. | ||
| - `VLLM_GRAPH_DECODE_STRATEGY`: strategy determining order of decode graph capture, `min_tokens` or `max_bs`. The default is `max_bs`. | ||
| - `VLLM_EXPONENTIAL_BUCKETING`: if `true`, enables exponential bucket spacing instead of linear. The default is `true`. | ||
| - `VLLM_PROMPT_BS_BUCKET_MAX`: prefill batch size max. The default is `1`. |
There was a problem hiding this comment.
Should we add a recomendation that VLLM_PROMPT_BS_BUCKET_MAX should be no bigger than max_num_batched_tokens as per issue seen in #331?
There was a problem hiding this comment.
Thanks. Updated
Signed-off-by: Iryna Boiko <[email protected]>
Signed-off-by: Iryna Boiko <[email protected]>
Signed-off-by: Iryna Boiko <[email protected]>
Signed-off-by: Iryna Boiko <[email protected]>
| - `VLLM_GRAPH_DECODE_STRATEGY`: strategy determining order of decode graph capture, `min_tokens` or `max_bs`. The default is `max_bs`. | ||
| - `VLLM_EXPONENTIAL_BUCKETING`: if `true`, enables exponential bucket spacing instead of linear. The default is `true`. | ||
| - `VLLM_{phase}_{dim}_BUCKET_{param}`: collection of 12 environment variables configuring ranges of bucketing mechanism (linear bucketing only). | ||
| - `VLLM_PROMPT_BS_BUCKET_MAX`: `(VLLM_PROMPT_BS_BUCKET_MAX * query) <=max_num_batched_tokens`- prefill batch size max. The default is`1`. |
There was a problem hiding this comment.
please resolve conflicts and we can merge
| - `VLLM_GRAPH_DECODE_STRATEGY`: strategy determining order of decode graph capture, `min_tokens` or `max_bs`. The default is `max_bs`. | ||
| - `VLLM_EXPONENTIAL_BUCKETING`: if `true`, enables exponential bucket spacing instead of linear. The default is `true`. | ||
| - `VLLM_{phase}_{dim}_BUCKET_{param}`: collection of 12 environment variables configuring ranges of bucketing mechanism (linear bucketing only). | ||
| - `VLLM_PROMPT_BS_BUCKET_MAX`: `(VLLM_PROMPT_BS_BUCKET_MAX * query) <=max_num_batched_tokens`- prefill batch size max. The default is`1`. |
There was a problem hiding this comment.
What is query here? Is query the input seq length?
When default is 1, max_num_batched_tokens < query (chunked prefill) is allowed and often used mode.
Does this breaks chunked prefill (chunk being smaller than query) and all advantages of it?
6 participants
…inear warmup