[Usage]: openbmb-MiniCPM-Llama3-V-2_5 在vllm上无法多gpu使用

[renjingneng]

Your current environment

报错：RuntimeError: Expected all tensors to be on the same device, but found at least two devices, cuda:0 and cuda:1! (when checking argument for argument weight in method wrapper_CUDA__cudnn_convolution)

(VllmWorkerProcess pid=1470366) ERROR 07-26 13:31:20 multiproc_worker_utils.py:226] return forward_call(*args, **kwargs)
(VllmWorkerProcess pid=1470366) ERROR 07-26 13:31:20 multiproc_worker_utils.py:226] File "/root/miniconda3/envs/minicpmv/lib/python3.10/site-packages/transformers/models/idefics2/modeling_idefics2.py", line 617, in forward
(VllmWorkerProcess pid=1470366) ERROR 07-26 13:31:20 multiproc_worker_utils.py:226] hidden_states = self.embeddings(pixel_values=pixel_values, patch_attention_mask=patch_attention_mask)
(VllmWorkerProcess pid=1470366) ERROR 07-26 13:31:20 multiproc_worker_utils.py:226] File "/root/miniconda3/envs/minicpmv/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
(VllmWorkerProcess pid=1470366) ERROR 07-26 13:31:20 multiproc_worker_utils.py:226] return self._call_impl(*args, **kwargs)
(VllmWorkerProcess pid=1470366) ERROR 07-26 13:31:20 multiproc_worker_utils.py:226] File "/root/miniconda3/envs/minicpmv/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
(VllmWorkerProcess pid=1470366) ERROR 07-26 13:31:20 multiproc_worker_utils.py:226] return forward_call(*args, **kwargs)
(VllmWorkerProcess pid=1470366) ERROR 07-26 13:31:20 multiproc_worker_utils.py:226] File "/root/miniconda3/envs/minicpmv/lib/python3.10/site-packages/transformers/models/idefics2/modeling_idefics2.py", line 162, in forward
(VllmWorkerProcess pid=1470366) ERROR 07-26 13:31:20 multiproc_worker_utils.py:226] patch_embeds = self.patch_embedding(pixel_values)
(VllmWorkerProcess pid=1470366) ERROR 07-26 13:31:20 multiproc_worker_utils.py:226] File "/root/miniconda3/envs/minicpmv/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
(VllmWorkerProcess pid=1470366) ERROR 07-26 13:31:20 multiproc_worker_utils.py:226] return self._call_impl(*args, **kwargs)
(VllmWorkerProcess pid=1470366) ERROR 07-26 13:31:20 multiproc_worker_utils.py:226] File "/root/miniconda3/envs/minicpmv/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
(VllmWorkerProcess pid=1470366) ERROR 07-26 13:31:20 multiproc_worker_utils.py:226] return forward_call(*args, **kwargs)
(VllmWorkerProcess pid=1470366) ERROR 07-26 13:31:20 multiproc_worker_utils.py:226] File "/root/miniconda3/envs/minicpmv/lib/python3.10/site-packages/torch/nn/modules/conv.py", line 460, in forward
(VllmWorkerProcess pid=1470366) ERROR 07-26 13:31:20 multiproc_worker_utils.py:226] return self._conv_forward(input, self.weight, self.bias)
(VllmWorkerProcess pid=1470366) ERROR 07-26 13:31:20 multiproc_worker_utils.py:226] File "/root/miniconda3/envs/minicpmv/lib/python3.10/site-packages/torch/nn/modules/conv.py", line 456, in _conv_forward
(VllmWorkerProcess pid=1470366) ERROR 07-26 13:31:20 multiproc_worker_utils.py:226] return F.conv2d(input, weight, bias, self.stride,
(VllmWorkerProcess pid=1470366) ERROR 07-26 13:31:20 multiproc_worker_utils.py:226] RuntimeError: Expected all tensors to be on the same device, but found at least two devices, cuda:0 and cuda:1! (when checking argument for argument weight in method wrapper_CUDA__cudnn_convolution)
(VllmWorkerProcess pid=1470366) ERROR 07-26 13:31:20 multiproc_worker_utils.py:226]
^C^C^C^C^C/root/miniconda3/envs/minicpmv/lib/python3.10/multiprocessing/resource_tracker.py:224: UserWarning: resource_tracker: There appear to be 1 leaked shared_memory objects to clean up at shutdown
warnings.warn('resource_tracker: There appear to be %d '
Killed

How would you like to use vllm

No response

[renjingneng]

补充：这个命令 vllm serve /aixunlian/renjingneng/tasks/task_6/model/openbmb-MiniCPM-Llama3-V-2_5 --port 8010 --trust-remote-code --tensor-parallel-size 2 --max_model_len 2000 报错

[HwwwwwwwH]

补充：这个命令 vllm serve /aixunlian/renjingneng/tasks/task_6/model/openbmb-MiniCPM-Llama3-V-2_5 --port 8010 --trust-remote-code --tensor-parallel-size 2 --max_model_len 2000 报错

I'll check this. BTW, is this issue #9 still exists?