[Frontend] Warn if user max_model_len is greater than derived max_model_len

[fialhocoelho]

Summary

Switch the error condition to a warning in cases where the user-specified max_model_len exceeds the derived max_model_len from the model's configuration parameters. This adjustment acknowledges that users may need to set values higher than those defined in the model's configuration file to meet specific requirements.

Motivation

Previously, an error was triggered when user max_model_len exceeded the derived value, potentially leading to unintended behavior or CUDA errors. By changing this to a warning, users are alerted without halting execution, allowing flexibility depending on their needs.

Notes

• Adjusted from f-string interpolation to % formatting for logging, in line with best practices (referencing W1203).
• The change ensures compatibility with environments where logging-format-interpolation may be disabled, maintaining robustness across different configurations.

[maxdebayser]

LGTM

[Yard1]

I am worried this warning will go unnoticed. I would prefer an exception. If we need an escape hatch here for some reason, then we should default to an exception and only override that behavior with an environment variable.

[njhill]

@Yard1 in some ways the user would already be intentionally overriding the model default here via explicitly passing --max-model-length. Do you think that's sufficient?

[robertgshaw2-neuralmagic]

We currently have a similar issue with ignored warnings when users deploy the chat/completions api for models without a chat template

I agree with Antoni here --- I think an env variable would make sense

[njhill]

@fialhocoelho could you make this change? Check https://github.com/vllm-project/vllm/blob/main/vllm/envs.py to see how other env vars are handled. Perhaps we could call it something like VLLM_ALLOW_LONG_MAX_MODEL_LEN

[fialhocoelho]

@fialhocoelho could you make this change? Check https://github.com/vllm-project/vllm/blob/main/vllm/envs.py to see how other env vars are handled. Perhaps we could call it something like VLLM_ALLOW_LONG_MAX_MODEL_LEN

Sure, @njhill . Thanks for the references. I'll start making the changes right away.

[fialhocoelho]

Explanation of Changes

This update addresses the handling of user-specified MAX_SEQUENCE_LENGTH values that exceed the model's maximum sequence length. The behavior is controlled by the environment variable VLLM_ALLOW_LONG_MAX_MODEL_LEN, which defaults to 0.

• Default Behavior (VLLM_ALLOW_LONG_MAX_MODEL_LEN=0):

  • If the user-specified MAX_SEQUENCE_LENGTH is greater than the model's maximum sequence length, an error will be raised. This prevents potential incorrect model outputs or CUDA errors that could arise from exceeding the model's capabilities.

• Optional Behavior (VLLM_ALLOW_LONG_MAX_MODEL_LEN=1):

  • If VLLM_ALLOW_LONG_MAX_MODEL_LEN is set to 1, the system will allow the user-specified MAX_SEQUENCE_LENGTH to exceed the model's maximum. Instead of raising an error, a warning message will be logged, advising users to ensure the value is correct and within the model's context size. This provides flexibility for advanced users who understand the risks and need to override the default limit.

[fialhocoelho]

Convert to draft to test with latest upstream version.

[fialhocoelho]

Tested with the latest upstream build image, and it works properly. Ready for review.