vllm issue: image_grid_thw[image_index][0], list index out of range #16
Description
Hi authors, thanks for your great work. I want to reproduce the web UI demo part, but I'm encountering an index error. The version of vllm is 0.7.3
The full info is as follows:
> CUDA_VISIBLE_DEVICES=1 vllm serve ./qwen2.5-vl-7b-instruct --served-model-name qwen2.5-vl-7b-instruct --host 127.0.0.1 --port 8000 --tensor-parallel-size 1 --trust-remote-code
INFO 09-28 08:16:03 __init__.py:207] Automatically detected platform cuda.
INFO 09-28 08:16:03 api_server.py:912] vLLM API server version 0.7.3
INFO 09-28 08:16:03 api_server.py:913] args: Namespace(subparser='serve', model_tag='./qwen2.5-vl-7b-instruct', config='', host='127.0.0.1', port=8000, uvicorn_log_level='info', allow_credentials=False, allowed_origins=['*'], allowed_methods=['*'], allowed_headers=['*'], api_key=None, lora_modules=None, prompt_adapters=None, chat_template=None, chat_template_content_format='auto', response_role='assistant', ssl_keyfile=None, ssl_certfile=None, ssl_ca_certs=None, ssl_cert_reqs=0, root_path=None, middleware=[], return_tokens_as_token_ids=False, disable_frontend_multiprocessing=False, enable_request_id_headers=False, enable_auto_tool_choice=False, enable_reasoning=False, reasoning_parser=None, tool_call_parser=None, tool_parser_plugin='', model='./qwen2.5-vl-7b-instruct', task='auto', tokenizer=None, skip_tokenizer_init=False, revision=None, code_revision=None, tokenizer_revision=None, tokenizer_mode='auto', trust_remote_code=True, allowed_local_media_path=None, download_dir=None, load_format='auto', config_format=<ConfigFormat.AUTO: 'auto'>, dtype='auto', kv_cache_dtype='auto', max_model_len=None, guided_decoding_backend='xgrammar', logits_processor_pattern=None, model_impl='auto', distributed_executor_backend=None, pipeline_parallel_size=1, tensor_parallel_size=1, max_parallel_loading_workers=None, ray_workers_use_nsight=False, block_size=None, enable_prefix_caching=None, disable_sliding_window=False, use_v2_block_manager=True, num_lookahead_slots=0, seed=0, swap_space=4, cpu_offload_gb=0, gpu_memory_utilization=0.9, num_gpu_blocks_override=None, max_num_batched_tokens=None, max_num_partial_prefills=1, max_long_partial_prefills=1, long_prefill_token_threshold=0, max_num_seqs=None, max_logprobs=20, disable_log_stats=False, quantization=None, rope_scaling=None, rope_theta=None, hf_overrides=None, enforce_eager=False, max_seq_len_to_capture=8192, disable_custom_all_reduce=False, tokenizer_pool_size=0, tokenizer_pool_type='ray', tokenizer_pool_extra_config=None, limit_mm_per_prompt=None, mm_processor_kwargs=None, disable_mm_preprocessor_cache=False, enable_lora=False, enable_lora_bias=False, max_loras=1, max_lora_rank=16, lora_extra_vocab_size=256, lora_dtype='auto', long_lora_scaling_factors=None, max_cpu_loras=None, fully_sharded_loras=False, enable_prompt_adapter=False, max_prompt_adapters=1, max_prompt_adapter_token=0, device='auto', num_scheduler_steps=1, multi_step_stream_outputs=True, scheduler_delay_factor=0.0, enable_chunked_prefill=None, speculative_model=None, speculative_model_quantization=None, num_speculative_tokens=None, speculative_disable_mqa_scorer=False, speculative_draft_tensor_parallel_size=None, speculative_max_model_len=None, speculative_disable_by_batch_size=None, ngram_prompt_lookup_max=None, ngram_prompt_lookup_min=None, spec_decoding_acceptance_method='rejection_sampler', typical_acceptance_sampler_posterior_threshold=None, typical_acceptance_sampler_posterior_alpha=None, disable_logprobs_during_spec_decoding=None, model_loader_extra_config=None, ignore_patterns=[], preemption_mode=None, served_model_name=['qwen2.5-vl-7b-instruct'], qlora_adapter_name_or_path=None, otlp_traces_endpoint=None, collect_detailed_traces=None, disable_async_output_proc=False, scheduling_policy='fcfs', scheduler_cls='vllm.core.scheduler.Scheduler', override_neuron_config=None, override_pooler_config=None, compilation_config=None, kv_transfer_config=None, worker_cls='auto', generation_config=None, override_generation_config=None, enable_sleep_mode=False, calculate_kv_scales=False, additional_config=None, disable_log_requests=False, max_log_len=None, disable_fastapi_docs=False, enable_prompt_tokens_details=False, dispatch_function=<function ServeSubcommand.cmd at 0x7fce98f61c60>)
INFO 09-28 08:16:03 api_server.py:209] Started engine process with PID 3667826
`torch_dtype` is deprecated! Use `dtype` instead!
INFO 09-28 08:16:07 __init__.py:207] Automatically detected platform cuda.
`torch_dtype` is deprecated! Use `dtype` instead!
INFO 09-28 08:16:08 config.py:549] This model supports multiple tasks: {'embed', 'generate', 'score', 'reward', 'classify'}. Defaulting to 'generate'.
WARNING 09-28 08:16:08 arg_utils.py:1197] The model has a long context length (128000). This may cause OOM errors during the initial memory profiling phase, or result in low performance due to small KV cache space. Consider setting --max-model-len to a smaller value.
INFO 09-28 08:16:12 config.py:549] This model supports multiple tasks: {'generate', 'reward', 'classify', 'score', 'embed'}. Defaulting to 'generate'.
WARNING 09-28 08:16:12 arg_utils.py:1197] The model has a long context length (128000). This may cause OOM errors during the initial memory profiling phase, or result in low performance due to small KV cache space. Consider setting --max-model-len to a smaller value.
INFO 09-28 08:16:12 llm_engine.py:234] Initializing a V0 LLM engine (v0.7.3) with config: model='./qwen2.5-vl-7b-instruct', speculative_config=None, tokenizer='./qwen2.5-vl-7b-instruct', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, override_neuron_config=None, tokenizer_revision=None, trust_remote_code=True, dtype=torch.bfloat16, max_seq_len=128000, download_dir=None, load_format=LoadFormat.AUTO, tensor_parallel_size=1, pipeline_parallel_size=1, disable_custom_all_reduce=False, quantization=None, enforce_eager=False, kv_cache_dtype=auto, device_config=cuda, decoding_config=DecodingConfig(guided_decoding_backend='xgrammar'), observability_config=ObservabilityConfig(otlp_traces_endpoint=None, collect_model_forward_time=False, collect_model_execute_time=False), seed=0, served_model_name=qwen2.5-vl-7b-instruct, num_scheduler_steps=1, multi_step_stream_outputs=True, enable_prefix_caching=False, chunked_prefill_enabled=False, use_async_output_proc=True, disable_mm_preprocessor_cache=False, mm_processor_kwargs=None, pooler_config=None, compilation_config={"splitting_ops":[],"compile_sizes":[],"cudagraph_capture_sizes":[256,248,240,232,224,216,208,200,192,184,176,168,160,152,144,136,128,120,112,104,96,88,80,72,64,56,48,40,32,24,16,8,4,2,1],"max_capture_size":256}, use_cached_outputs=True,
INFO 09-28 08:16:14 cuda.py:229] Using Flash Attention backend.
INFO 09-28 08:16:14 model_runner.py:1110] Starting to load model ./qwen2.5-vl-7b-instruct...
WARNING 09-28 08:16:14 vision.py:94] Current `vllm-flash-attn` has a bug inside vision module, so we use xformers backend instead. You can run `pip install flash-attn` to use flash-attention backend.
INFO 09-28 08:16:14 config.py:3054] cudagraph sizes specified by model runner [1, 2, 4, 8, 16, 24, 32, 40, 48, 56, 64, 72, 80, 88, 96, 104, 112, 120, 128, 136, 144, 152, 160, 168, 176, 184, 192, 200, 208, 216, 224, 232, 240, 248, 256] is overridden by config [256, 128, 2, 1, 4, 136, 8, 144, 16, 152, 24, 160, 32, 168, 40, 176, 48, 184, 56, 192, 64, 200, 72, 208, 80, 216, 88, 120, 224, 96, 232, 104, 240, 112, 248]
Loading safetensors checkpoint shards: 0% Completed | 0/5 [00:00<?, ?it/s]
Loading safetensors checkpoint shards: 20% Completed | 1/5 [00:00<00:02, 1.70it/s]
Loading safetensors checkpoint shards: 40% Completed | 2/5 [00:01<00:01, 1.54it/s]
Loading safetensors checkpoint shards: 60% Completed | 3/5 [00:01<00:01, 1.55it/s]
Loading safetensors checkpoint shards: 80% Completed | 4/5 [00:02<00:00, 1.58it/s]
Loading safetensors checkpoint shards: 100% Completed | 5/5 [00:02<00:00, 2.07it/s]
Loading safetensors checkpoint shards: 100% Completed | 5/5 [00:02<00:00, 1.82it/s]
INFO 09-28 08:16:17 model_runner.py:1115] Loading model weights took 15.6270 GB
The image processor of type `Qwen2VLImageProcessor` is now loaded as a fast processor by default, even if the model checkpoint was saved with a slow processor. This is a breaking change and may produce slightly different outputs. To continue using the slow processor, instantiate this class with `use_fast=False`. Note that this behavior will be extended to all models in a future release.
INFO 09-28 08:17:05 worker.py:267] Memory profiling takes 47.90 seconds
INFO 09-28 08:17:05 worker.py:267] the current vLLM instance can use total_gpu_memory (79.15GiB) x gpu_memory_utilization (0.90) = 71.24GiB
INFO 09-28 08:17:05 worker.py:267] model weights take 15.63GiB; non_torch_memory takes 0.09GiB; PyTorch activation peak memory takes 21.27GiB; the rest of the memory reserved for KV Cache is 34.25GiB.
INFO 09-28 08:17:06 executor_base.py:111] # cuda blocks: 40077, # CPU blocks: 4681
INFO 09-28 08:17:06 executor_base.py:116] Maximum concurrency for 128000 tokens per request: 5.01x
INFO 09-28 08:17:09 model_runner.py:1434] Capturing cudagraphs for decoding. This may lead to unexpected consequences if the model is not static. To run the model in eager mode, set 'enforce_eager=True' or use '--enforce-eager' in the CLI. If out-of-memory error occurs during cudagraph capture, consider decreasing `gpu_memory_utilization` or switching to eager mode. You can also reduce the `max_num_seqs` as needed to decrease memory usage.
Capturing CUDA graph shapes: 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 35/35 [00:12<00:00, 2.77it/s]
INFO 09-28 08:17:22 model_runner.py:1562] Graph capturing finished in 13 secs, took 1.91 GiB
INFO 09-28 08:17:22 llm_engine.py:436] init engine (profile, create kv cache, warmup model) took 64.59 seconds
INFO 09-28 08:17:22 api_server.py:958] Starting vLLM API server on http://127.0.0.1:8000
INFO 09-28 08:17:22 launcher.py:23] Available routes are:
INFO 09-28 08:17:22 launcher.py:31] Route: /openapi.json, Methods: HEAD, GET
INFO 09-28 08:17:22 launcher.py:31] Route: /docs, Methods: HEAD, GET
INFO 09-28 08:17:22 launcher.py:31] Route: /docs/oauth2-redirect, Methods: HEAD, GET
INFO 09-28 08:17:22 launcher.py:31] Route: /redoc, Methods: HEAD, GET
INFO 09-28 08:17:22 launcher.py:31] Route: /health, Methods: GET
INFO 09-28 08:17:22 launcher.py:31] Route: /ping, Methods: POST, GET
INFO 09-28 08:17:22 launcher.py:31] Route: /tokenize, Methods: POST
INFO 09-28 08:17:22 launcher.py:31] Route: /detokenize, Methods: POST
INFO 09-28 08:17:22 launcher.py:31] Route: /v1/models, Methods: GET
INFO 09-28 08:17:22 launcher.py:31] Route: /version, Methods: GET
INFO 09-28 08:17:22 launcher.py:31] Route: /v1/chat/completions, Methods: POST
INFO 09-28 08:17:22 launcher.py:31] Route: /v1/completions, Methods: POST
INFO 09-28 08:17:22 launcher.py:31] Route: /v1/embeddings, Methods: POST
INFO 09-28 08:17:22 launcher.py:31] Route: /pooling, Methods: POST
INFO 09-28 08:17:22 launcher.py:31] Route: /score, Methods: POST
INFO 09-28 08:17:22 launcher.py:31] Route: /v1/score, Methods: POST
INFO 09-28 08:17:22 launcher.py:31] Route: /v1/audio/transcriptions, Methods: POST
INFO 09-28 08:17:22 launcher.py:31] Route: /rerank, Methods: POST
INFO 09-28 08:17:22 launcher.py:31] Route: /v1/rerank, Methods: POST
INFO 09-28 08:17:22 launcher.py:31] Route: /v2/rerank, Methods: POST
INFO 09-28 08:17:22 launcher.py:31] Route: /invocations, Methods: POST
INFO: Started server process [3667636]
INFO: Waiting for application startup.
INFO: Application startup complete.
INFO: 127.0.0.1:50638 - "GET /v1/models HTTP/1.1" 200 OK
INFO 09-28 08:20:26 chat_utils.py:332] Detected the chat template content format to be 'openai'. You can set `--chat-template-content-format` to override this.
INFO 09-28 08:20:26 logger.py:39] Received request chatcmpl-62731522ef1d486484bf3a902fce2039: prompt: '<|im_start|>system\nYou are using a Mobile device. You are able to use a Action Space Operator to interact with the mobile based on the given task and screenshot.\n\n## Action Space\nYour available "Next Action" only include:\n- click(point=[x,y]): Click on the coordinate point specified on the screen (x,y).\n- long_press(point=[x,y]): Long press the screen to specify coordinates (x,y).\n- type(text=\'hello world\'): Types a string of text.\n- scroll(start_point=[x1,y1], end_point=[x2,y2]): Scroll the screen, (x1,y1) is the starting coordinate position, (x2,y2) is the end coordinate position. In particular, when y1=y2, you can swipe left and right on the desktop to switch pages, which is very helpful for finding a specific application.\n- press_home(): Back to Home page.\n- press_back(): Back to previous page.\n- finished(answer=\'\'): Submit the task regardless of whether it succeeds or fails. The answer parameter is to summarize the content of the reply to the user.\n- call_user(question=\'\'): Submit the task and call the user when the task is unsolvable, or when you need the user\'s help.\n- wait(): Wait for loading to complete.\n\n## Note\n- Action click, long_press and scroll must contain coordinates within.\n- You may be given some history plan and actions, this is the response from the previous loop.\n- You should carefully consider your plan base on the task, screenshot, and history actions.\n- Write a small plan and finally summarize your next action (with its target element) in one sentence in `Thought` part.\n\n## Suggestions\n- If you need to open an APP, when the home page is not available, you can scroll down to the search page to find the corresponding APP.\n- When the screen of the previous operation is not responsive, you need to avoid performing the same action in the next step.\n- Shopping or life services apps, you should make use of the in-app search function as much as possible to find quickly.\n- Reduce the execution steps as much as possible, and find the optimal execution path to achieve the task goal.\n\n## Format\nTask: The task description.\nObservation: The mobile screenshot or user response.\nThought: The process of thinking.\nAction: The next action. Must be one of the Action Space.\n\n**Be aware that Observation, Thought, and Action will be repeated.**\n\nNow, let\'s begin!<|im_end|>\n<|im_start|>user\nTask: open wechatObservation: <|vision_start|><|image_pad|><|vision_end|><|vision_start|><|image_pad|><|vision_end|><|im_end|>\n<|im_start|>assistant\n', params: SamplingParams(n=1, presence_penalty=0.0, frequency_penalty=0.0, repetition_penalty=1.0, temperature=0.3, top_p=1.0, top_k=-1, min_p=0.0, seed=None, stop=['Observation'], stop_token_ids=[], bad_words=[], include_stop_str_in_output=False, ignore_eos=False, max_tokens=1024, min_tokens=0, logprobs=None, prompt_logprobs=None, skip_special_tokens=True, spaces_between_special_tokens=True, truncate_prompt_tokens=None, guided_decoding=None), prompt_token_ids: None, lora_request: None, prompt_adapter_request: None.
INFO 09-28 08:20:28 engine.py:280] Added request chatcmpl-62731522ef1d486484bf3a902fce2039.
CRITICAL 09-28 08:20:28 launcher.py:104] MQLLMEngine is already dead, terminating server process
INFO: 127.0.0.1:57966 - "POST /v1/chat/completions HTTP/1.1" 500 Internal Server Error
ERROR 09-28 08:20:28 engine.py:140] IndexError('list index out of range')
ERROR 09-28 08:20:28 engine.py:140] Traceback (most recent call last):
ERROR 09-28 08:20:28 engine.py:140] File "/home/xxx/micromamba/envs/vllm/lib/python3.12/site-packages/vllm/engine/multiprocessing/engine.py", line 138, in start
ERROR 09-28 08:20:28 engine.py:140] self.run_engine_loop()
ERROR 09-28 08:20:28 engine.py:140] File "/home/xxx/micromamba/envs/vllm/lib/python3.12/site-packages/vllm/engine/multiprocessing/engine.py", line 201, in run_engine_loop
ERROR 09-28 08:20:28 engine.py:140] request_outputs = self.engine_step()
ERROR 09-28 08:20:28 engine.py:140] ^^^^^^^^^^^^^^^^^^
ERROR 09-28 08:20:28 engine.py:140] File "/home/xxx/micromamba/envs/vllm/lib/python3.12/site-packages/vllm/engine/multiprocessing/engine.py", line 219, in engine_step
ERROR 09-28 08:20:28 engine.py:140] raise e
ERROR 09-28 08:20:28 engine.py:140] File "/home/xxx/micromamba/envs/vllm/lib/python3.12/site-packages/vllm/engine/multiprocessing/engine.py", line 210, in engine_step
ERROR 09-28 08:20:28 engine.py:140] return self.engine.step()
ERROR 09-28 08:20:28 engine.py:140] ^^^^^^^^^^^^^^^^^^
ERROR 09-28 08:20:28 engine.py:140] File "/home/xxx/micromamba/envs/vllm/lib/python3.12/site-packages/vllm/engine/llm_engine.py", line 1391, in step
ERROR 09-28 08:20:28 engine.py:140] outputs = self.model_executor.execute_model(
ERROR 09-28 08:20:28 engine.py:140] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ERROR 09-28 08:20:28 engine.py:140] File "/home/xxx/micromamba/envs/vllm/lib/python3.12/site-packages/vllm/executor/executor_base.py", line 139, in execute_model
ERROR 09-28 08:20:28 engine.py:140] output = self.collective_rpc("execute_model",
ERROR 09-28 08:20:28 engine.py:140] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ERROR 09-28 08:20:28 engine.py:140] File "/home/xxx/micromamba/envs/vllm/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 56, in collective_rpc
ERROR 09-28 08:20:28 engine.py:140] answer = run_method(self.driver_worker, method, args, kwargs)
ERROR 09-28 08:20:28 engine.py:140] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ERROR 09-28 08:20:28 engine.py:140] File "/home/xxx/micromamba/envs/vllm/lib/python3.12/site-packages/vllm/utils.py", line 2196, in run_method
ERROR 09-28 08:20:28 engine.py:140] return func(*args, **kwargs)
ERROR 09-28 08:20:28 engine.py:140] ^^^^^^^^^^^^^^^^^^^^^
ERROR 09-28 08:20:28 engine.py:140] File "/home/xxx/micromamba/envs/vllm/lib/python3.12/site-packages/vllm/worker/worker_base.py", line 394, in execute_model
ERROR 09-28 08:20:28 engine.py:140] inputs = self.prepare_input(execute_model_req)
ERROR 09-28 08:20:28 engine.py:140] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ERROR 09-28 08:20:28 engine.py:140] File "/home/xxx/micromamba/envs/vllm/lib/python3.12/site-packages/vllm/worker/worker_base.py", line 379, in prepare_input
ERROR 09-28 08:20:28 engine.py:140] return self._get_driver_input_and_broadcast(execute_model_req)
ERROR 09-28 08:20:28 engine.py:140] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ERROR 09-28 08:20:28 engine.py:140] File "/home/xxx/micromamba/envs/vllm/lib/python3.12/site-packages/vllm/worker/worker_base.py", line 341, in _get_driver_input_and_broadcast
ERROR 09-28 08:20:28 engine.py:140] self.model_runner.prepare_model_input(
ERROR 09-28 08:20:28 engine.py:140] File "/home/xxx/micromamba/envs/vllm/lib/python3.12/site-packages/vllm/worker/model_runner.py", line 1628, in prepare_model_input
ERROR 09-28 08:20:28 engine.py:140] model_input = self._prepare_model_input_tensors(
ERROR 09-28 08:20:28 engine.py:140] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ERROR 09-28 08:20:28 engine.py:140] File "/home/xxx/micromamba/envs/vllm/lib/python3.12/site-packages/vllm/worker/model_runner.py", line 1216, in _prepare_model_input_tensors
ERROR 09-28 08:20:28 engine.py:140] self.builder.add_seq_group(seq_group_metadata)
ERROR 09-28 08:20:28 engine.py:140] File "/home/xxx/micromamba/envs/vllm/lib/python3.12/site-packages/vllm/worker/model_runner.py", line 760, in add_seq_group
ERROR 09-28 08:20:28 engine.py:140] per_seq_group_fn(inter_data, seq_group_metadata)
ERROR 09-28 08:20:28 engine.py:140] File "/home/xxx/micromamba/envs/vllm/lib/python3.12/site-packages/vllm/worker/model_runner.py", line 715, in _compute_multi_modal_input
ERROR 09-28 08:20:28 engine.py:140] MRotaryEmbedding.get_input_positions(
ERROR 09-28 08:20:28 engine.py:140] File "/home/xxx/micromamba/envs/vllm/lib/python3.12/site-packages/vllm/model_executor/layers/rotary_embedding.py", line 854, in get_input_positions
ERROR 09-28 08:20:28 engine.py:140] MRotaryEmbedding.get_input_positions_tensor(
ERROR 09-28 08:20:28 engine.py:140] File "/home/xxx/micromamba/envs/vllm/lib/python3.12/site-packages/vllm/model_executor/layers/rotary_embedding.py", line 914, in get_input_positions_tensor
ERROR 09-28 08:20:28 engine.py:140] image_grid_thw[image_index][0],
ERROR 09-28 08:20:28 engine.py:140] ~~~~~~~~~~~~~~^^^^^^^^^^^^^
ERROR 09-28 08:20:28 engine.py:140] IndexError: list index out of range
INFO: Shutting down
INFO: Waiting for application shutdown.
INFO: Application shutdown complete.
INFO: Finished server process [3667636]
I have tested that qwen is successfully deployed via vllm:
curl http://127.0.0.1:8000/v1/models
{"object":"list","data":[{"id":"qwen2.5-vl-7b-instruct","object":"model","created":1759047522,"owned_by":"vllm","root":"./qwen2.5-vl-7b-instruct","parent":null,"max_model_len":128000,"permission":[{"id":"modelperm-f6b8e5164b254a14932e215cfa579f66","object":"model_permission","created":1759047522,"allow_create_engine":false,"allow_sampling":true,"allow_logprobs":true,"allow_search_indices":false,"allow_view":true,"allow_fine_tuning":false,"organization":"*","group":null,"is_blocking":false}]}]}%
I have searched for other possibly related issues but I cannot find a proper solution. I wonder if this is some prompt issue with vllm or qwen? Could you please offer some help? Thanks for your reply in advance.