Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Bug]: stack trace for "Watchdog caught collective operation timeout" #12625

Open
1 task done
remusao opened this issue Jan 31, 2025 · 0 comments
Open
1 task done

[Bug]: stack trace for "Watchdog caught collective operation timeout" #12625

remusao opened this issue Jan 31, 2025 · 0 comments
Labels
bug Something isn't working

Comments

@remusao
Copy link

remusao commented Jan 31, 2025 •
edited
Loading

Your current environment

The output of `python collect_env.py` 
INFO 01-31 10:13:08 __init__.py:183] Automatically detected platform cuda.
Collecting environment information...
PyTorch version: 2.5.1+cu124
Is debug build: False
CUDA used to build PyTorch: 12.4
ROCM used to build PyTorch: N/A

OS: Ubuntu 22.04.3 LTS (x86_64)
GCC version: (Ubuntu 11.4.0-1ubuntu1~22.04) 11.4.0
Clang version: Could not collect
CMake version: Could not collect
Libc version: glibc-2.35

Python version: 3.12.8 (main, Dec  4 2024, 08:54:12) [GCC 11.4.0] (64-bit runtime)
Python platform: Linux-5.15.0-124-generic-x86_64-with-glibc2.35
Is CUDA available: True
CUDA runtime version: Could not collect
CUDA_MODULE_LOADING set to: LAZY
GPU models and configuration: 
GPU 0: NVIDIA H100 80GB HBM3
GPU 1: NVIDIA H100 80GB HBM3
GPU 2: NVIDIA H100 80GB HBM3
GPU 3: NVIDIA H100 80GB HBM3
GPU 4: NVIDIA H100 80GB HBM3
GPU 5: NVIDIA H100 80GB HBM3
GPU 6: NVIDIA H100 80GB HBM3
GPU 7: NVIDIA H100 80GB HBM3

Nvidia driver version: 550.90.07
cuDNN version: Could not collect
HIP runtime version: N/A
MIOpen runtime version: N/A
Is XNNPACK available: True

CPU:
Architecture:                         x86_64
CPU op-mode(s):                       32-bit, 64-bit
Address sizes:                        43 bits physical, 57 bits virtual
Byte Order:                           Little Endian
CPU(s):                               128
On-line CPU(s) list:                  0-127
Vendor ID:                            GenuineIntel
Model name:                           Intel(R) Xeon(R) Platinum 8468
CPU family:                           6
Model:                                143
Thread(s) per core:                   2
Core(s) per socket:                   32
Socket(s):                            2
Stepping:                             8
BogoMIPS:                             4200.00
Flags:                                fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ss ht syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon rep_good nopl xtopology cpuid tsc_known_freq pni pclmulqdq ssse3 fma cx16 pdcm pcid sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand hypervisor lahf_lm abm 3dnowprefetch cpuid_fault invpcid_single ssbd ibrs ibpb stibp ibrs_enhanced fsgsbase tsc_adjust bmi1 avx2 smep bmi2 erms invpcid avx512f avx512dq rdseed adx smap avx512ifma clflushopt clwb avx512cd sha_ni avx512bw avx512vl xsaveopt xsavec xgetbv1 xsaves avx_vnni avx512_bf16 wbnoinvd arat avx512vbmi umip pku ospke waitpkg avx512_vbmi2 gfni vaes vpclmulqdq avx512_vnni avx512_bitalg avx512_vpopcntdq la57 rdpid bus_lock_detect cldemote movdiri movdir64b fsrm md_clear serialize tsxldtrk avx512_fp16 arch_capabilities
Hypervisor vendor:                    KVM
Virtualization type:                  full
L1d cache:                            4 MiB (128 instances)
L1i cache:                            4 MiB (128 instances)
L2 cache:                             256 MiB (64 instances)
L3 cache:                             32 MiB (2 instances)
NUMA node(s):                         2
NUMA node0 CPU(s):                    0-63
NUMA node1 CPU(s):                    64-127
Vulnerability Gather data sampling:   Not affected
Vulnerability Itlb multihit:          Not affected
Vulnerability L1tf:                   Not affected
Vulnerability Mds:                    Not affected
Vulnerability Meltdown:               Not affected
Vulnerability Mmio stale data:        Unknown: No mitigations
Vulnerability Reg file data sampling: Not affected
Vulnerability Retbleed:               Not affected
Vulnerability Spec rstack overflow:   Not affected
Vulnerability Spec store bypass:      Mitigation; Speculative Store Bypass disabled via prctl and seccomp
Vulnerability Spectre v1:             Mitigation; usercopy/swapgs barriers and __user pointer sanitization
Vulnerability Spectre v2:             Mitigation; Enhanced / Automatic IBRS; IBPB conditional; RSB filling; PBRSB-eIBRS SW sequence; BHI SW loop, KVM SW loop
Vulnerability Srbds:                  Not affected
Vulnerability Tsx async abort:        Mitigation; TSX disabled

Versions of relevant libraries:
[pip3] flashinfer==0.1.6+cu121torch2.4
[pip3] numpy==1.26.4
[pip3] nvidia-cublas-cu12==12.4.5.8
[pip3] nvidia-cuda-cupti-cu12==12.4.127
[pip3] nvidia-cuda-nvrtc-cu12==12.4.127
[pip3] nvidia-cuda-runtime-cu12==12.4.127
[pip3] nvidia-cudnn-cu12==9.1.0.70
[pip3] nvidia-cufft-cu12==11.2.1.3
[pip3] nvidia-curand-cu12==10.3.5.147
[pip3] nvidia-cusolver-cu12==11.6.1.9
[pip3] nvidia-cusparse-cu12==12.3.1.170
[pip3] nvidia-ml-py==12.570.86
[pip3] nvidia-nccl-cu12==2.21.5
[pip3] nvidia-nvjitlink-cu12==12.4.127
[pip3] nvidia-nvtx-cu12==12.4.127
[pip3] pyzmq==26.2.0
[pip3] torch==2.5.1
[pip3] torchvision==0.20.1
[pip3] transformers==4.48.1
[pip3] triton==3.1.0
[conda] Could not collect
ROCM Version: Could not collect
Neuron SDK Version: N/A
vLLM Version: 0.7.0
vLLM Build Flags:
CUDA Archs: Not Set; ROCm: Disabled; Neuron: Disabled
GPU Topology:
GPU0	GPU1	GPU2	GPU3	GPU4	GPU5	GPU6	GPU7	CPU Affinity	NUMA Affinity	GPU NUMA ID
GPU0	 X 	NV18	NV18	NV18	NV18	NV18	NV18	NV18	0-63	0		N/A
GPU1	NV18	 X 	NV18	NV18	NV18	NV18	NV18	NV18	0-63	0		N/A
GPU2	NV18	NV18	 X 	NV18	NV18	NV18	NV18	NV18	0-63	0		N/A
GPU3	NV18	NV18	NV18	 X 	NV18	NV18	NV18	NV18	0-63	0		N/A
GPU4	NV18	NV18	NV18	NV18	 X 	NV18	NV18	NV18	64-127	1		N/A
GPU5	NV18	NV18	NV18	NV18	NV18	 X 	NV18	NV18	64-127	1		N/A
GPU6	NV18	NV18	NV18	NV18	NV18	NV18	 X 	NV18	64-127	1		N/A
GPU7	NV18	NV18	NV18	NV18	NV18	NV18	NV18	 X 	64-127	1		N/A

Legend:

  X    = Self
  SYS  = Connection traversing PCIe as well as the SMP interconnect between NUMA nodes (e.g., QPI/UPI)
  NODE = Connection traversing PCIe as well as the interconnect between PCIe Host Bridges within a NUMA node
  PHB  = Connection traversing PCIe as well as a PCIe Host Bridge (typically the CPU)
  PXB  = Connection traversing multiple PCIe bridges (without traversing the PCIe Host Bridge)
  PIX  = Connection traversing at most a single PCIe bridge
  NV#  = Connection traversing a bonded set of # NVLinks

NVIDIA_VISIBLE_DEVICES=all
NVIDIA_REQUIRE_CUDA=cuda>=12.1 brand=tesla,driver>=470,driver<471 brand=unknown,driver>=470,driver<471 brand=nvidia,driver>=470,driver<471 brand=nvidiartx,driver>=470,driver<471 brand=geforce,driver>=470,driver<471 brand=geforcertx,driver>=470,driver<471 brand=quadro,driver>=470,driver<471 brand=quadrortx,driver>=470,driver<471 brand=titan,driver>=470,driver<471 brand=titanrtx,driver>=470,driver<471 brand=tesla,driver>=525,driver<526 brand=unknown,driver>=525,driver<526 brand=nvidia,driver>=525,driver<526 brand=nvidiartx,driver>=525,driver<526 brand=geforce,driver>=525,driver<526 brand=geforcertx,driver>=525,driver<526 brand=quadro,driver>=525,driver<526 brand=quadrortx,driver>=525,driver<526 brand=titan,driver>=525,driver<526 brand=titanrtx,driver>=525,driver<526
NVIDIA_DRIVER_CAPABILITIES=compute,utility
VLLM_USAGE_SOURCE=production-docker-image
CUDA_VERSION=12.1.0
CUDA_VISIBLE_DEVICES=0,1,2,3
CUDA_VISIBLE_DEVICES=0,1,2,3
VLLM_FLASH_ATTN_VERSION=3
LD_LIBRARY_PATH=/usr/local/lib/python3.12/dist-packages/cv2/../../lib64:/usr/local/nvidia/lib:/usr/local/nvidia/lib64
NCCL_CUMEM_ENABLE=0
TORCHINDUCTOR_COMPILE_THREADS=1
CUDA_MODULE_LOADING=LAZY

Model Input Dumps

Unfortunately, the problematic inputs cannot be pickled for some reason, here is what happens after Error in model execution:

(VllmWorkerProcess pid=350) ERROR 01-29 20:00:47 multiproc_worker_utils.py:240] RuntimeError: Error in model execution: NCCL error: unhandled cuda error (run with NCCL_DEBUG=INFO for details)
(VllmWorkerProcess pid=351) INFO 01-29 20:00:47 model_runner_base.py:120] Writing input of failed execution to /tmp/err_execute_model_input_20250129-200047.pkl...
(VllmWorkerProcess pid=351) WARNING 01-29 20:00:47 model_runner_base.py:143] Failed to pickle inputs of failed execution: CUDA error: an illegal memory access was encountered
(VllmWorkerProcess pid=351) WARNING 01-29 20:00:47 model_runner_base.py:143] CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.
(VllmWorkerProcess pid=351) WARNING 01-29 20:00:47 model_runner_base.py:143] For debugging consider passing CUDA_LAUNCH_BLOCKING=1
(VllmWorkerProcess pid=351) WARNING 01-29 20:00:47 model_runner_base.py:143] Compile with `TORCH_USE_CUDA_DSA` to enable device-side assertions.
And here is an example of a full trace 
(VllmWorkerProcess pid=350) ERROR 01-29 20:00:47 multiproc_worker_utils.py:240] Exception in worker VllmWorkerProcess while processing method start_worker_execution_loop.
(VllmWorkerProcess pid=350) ERROR 01-29 20:00:47 multiproc_worker_utils.py:240] Traceback (most recent call last):
(VllmWorkerProcess pid=350) ERROR 01-29 20:00:47 multiproc_worker_utils.py:240]   File "/usr/local/lib/python3.12/dist-packages/vllm/worker/model_runner_base.py", line 116, in _wrapper
(VllmWorkerProcess pid=350) ERROR 01-29 20:00:47 multiproc_worker_utils.py:240]     return func(*args, **kwargs)
(VllmWorkerProcess pid=350) ERROR 01-29 20:00:47 multiproc_worker_utils.py:240]            ^^^^^^^^^^^^^^^^^^^^^
(VllmWorkerProcess pid=350) ERROR 01-29 20:00:47 multiproc_worker_utils.py:240]   File "/usr/local/lib/python3.12/dist-packages/vllm/worker/model_runner.py", line 1716, in execute_model
(VllmWorkerProcess pid=350) ERROR 01-29 20:00:47 multiproc_worker_utils.py:240]     hidden_or_intermediate_states = model_executable(
(VllmWorkerProcess pid=350) ERROR 01-29 20:00:47 multiproc_worker_utils.py:240]                                     ^^^^^^^^^^^^^^^^^
(VllmWorkerProcess pid=350) ERROR 01-29 20:00:47 multiproc_worker_utils.py:240]   File "/usr/local/lib/python3.12/dist-packages/torch/nn/modules/module.py", line 1736, in _wrapped_call_impl
(VllmWorkerProcess pid=350) ERROR 01-29 20:00:47 multiproc_worker_utils.py:240]     return self._call_impl(*args, **kwargs)
(VllmWorkerProcess pid=350) ERROR 01-29 20:00:47 multiproc_worker_utils.py:240]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(VllmWorkerProcess pid=350) ERROR 01-29 20:00:47 multiproc_worker_utils.py:240]   File "/usr/local/lib/python3.12/dist-packages/torch/nn/modules/module.py", line 1747, in _call_impl
(VllmWorkerProcess pid=350) ERROR 01-29 20:00:47 multiproc_worker_utils.py:240]     return forward_call(*args, **kwargs)
(VllmWorkerProcess pid=350) ERROR 01-29 20:00:47 multiproc_worker_utils.py:240]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(VllmWorkerProcess pid=350) ERROR 01-29 20:00:47 multiproc_worker_utils.py:240]   File "/usr/local/lib/python3.12/dist-packages/vllm/model_executor/models/llama.py", line 538, in forward
(VllmWorkerProcess pid=350) ERROR 01-29 20:00:47 multiproc_worker_utils.py:240]     model_output = self.model(input_ids, positions, kv_caches,
(VllmWorkerProcess pid=350) ERROR 01-29 20:00:47 multiproc_worker_utils.py:240]                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(VllmWorkerProcess pid=350) ERROR 01-29 20:00:47 multiproc_worker_utils.py:240]   File "/usr/local/lib/python3.12/dist-packages/vllm/compilation/decorators.py", line 170, in __call__
(VllmWorkerProcess pid=350) ERROR 01-29 20:00:47 multiproc_worker_utils.py:240]     return self.forward(*args, **kwargs)
(VllmWorkerProcess pid=350) ERROR 01-29 20:00:47 multiproc_worker_utils.py:240]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(VllmWorkerProcess pid=350) ERROR 01-29 20:00:47 multiproc_worker_utils.py:240]   File "/usr/local/lib/python3.12/dist-packages/vllm/model_executor/models/llama.py", line 363, in forward
(VllmWorkerProcess pid=350) ERROR 01-29 20:00:47 multiproc_worker_utils.py:240]     hidden_states, residual = layer(positions, hidden_states,
(VllmWorkerProcess pid=350) ERROR 01-29 20:00:47 multiproc_worker_utils.py:240]                               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(VllmWorkerProcess pid=350) ERROR 01-29 20:00:47 multiproc_worker_utils.py:240]   File "/usr/local/lib/python3.12/dist-packages/torch/nn/modules/module.py", line 1736, in _wrapped_call_impl
(VllmWorkerProcess pid=350) ERROR 01-29 20:00:47 multiproc_worker_utils.py:240]     return self._call_impl(*args, **kwargs)
(VllmWorkerProcess pid=350) ERROR 01-29 20:00:47 multiproc_worker_utils.py:240]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(VllmWorkerProcess pid=350) ERROR 01-29 20:00:47 multiproc_worker_utils.py:240]   File "/usr/local/lib/python3.12/dist-packages/torch/nn/modules/module.py", line 1747, in _call_impl
(VllmWorkerProcess pid=350) ERROR 01-29 20:00:47 multiproc_worker_utils.py:240]     return forward_call(*args, **kwargs)
(VllmWorkerProcess pid=350) ERROR 01-29 20:00:47 multiproc_worker_utils.py:240]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(VllmWorkerProcess pid=350) ERROR 01-29 20:00:47 multiproc_worker_utils.py:240]   File "/usr/local/lib/python3.12/dist-packages/vllm/model_executor/models/llama.py", line 277, in forward
(VllmWorkerProcess pid=350) ERROR 01-29 20:00:47 multiproc_worker_utils.py:240]     hidden_states = self.self_attn(positions=positions,
(VllmWorkerProcess pid=350) ERROR 01-29 20:00:47 multiproc_worker_utils.py:240]                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(VllmWorkerProcess pid=350) ERROR 01-29 20:00:47 multiproc_worker_utils.py:240]   File "/usr/local/lib/python3.12/dist-packages/torch/nn/modules/module.py", line 1736, in _wrapped_call_impl
(VllmWorkerProcess pid=350) ERROR 01-29 20:00:47 multiproc_worker_utils.py:240]     return self._call_impl(*args, **kwargs)
(VllmWorkerProcess pid=350) ERROR 01-29 20:00:47 multiproc_worker_utils.py:240]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(VllmWorkerProcess pid=350) ERROR 01-29 20:00:47 multiproc_worker_utils.py:240]   File "/usr/local/lib/python3.12/dist-packages/torch/nn/modules/module.py", line 1747, in _call_impl
(VllmWorkerProcess pid=350) ERROR 01-29 20:00:47 multiproc_worker_utils.py:240]     return forward_call(*args, **kwargs)
(VllmWorkerProcess pid=350) ERROR 01-29 20:00:47 multiproc_worker_utils.py:240]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(VllmWorkerProcess pid=350) ERROR 01-29 20:00:47 multiproc_worker_utils.py:240]   File "/usr/local/lib/python3.12/dist-packages/vllm/model_executor/models/llama.py", line 202, in forward
(VllmWorkerProcess pid=350) ERROR 01-29 20:00:47 multiproc_worker_utils.py:240]     output, _ = self.o_proj(attn_output)
(VllmWorkerProcess pid=350) ERROR 01-29 20:00:47 multiproc_worker_utils.py:240]                 ^^^^^^^^^^^^^^^^^^^^^^^^
(VllmWorkerProcess pid=350) ERROR 01-29 20:00:47 multiproc_worker_utils.py:240]   File "/usr/local/lib/python3.12/dist-packages/torch/nn/modules/module.py", line 1736, in _wrapped_call_impl
(VllmWorkerProcess pid=350) ERROR 01-29 20:00:47 multiproc_worker_utils.py:240]     return self._call_impl(*args, **kwargs)
(VllmWorkerProcess pid=350) ERROR 01-29 20:00:47 multiproc_worker_utils.py:240]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(VllmWorkerProcess pid=350) ERROR 01-29 20:00:47 multiproc_worker_utils.py:240]   File "/usr/local/lib/python3.12/dist-packages/torch/nn/modules/module.py", line 1747, in _call_impl
(VllmWorkerProcess pid=350) ERROR 01-29 20:00:47 multiproc_worker_utils.py:240]     return forward_call(*args, **kwargs)
(VllmWorkerProcess pid=350) ERROR 01-29 20:00:47 multiproc_worker_utils.py:240]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(VllmWorkerProcess pid=350) ERROR 01-29 20:00:47 multiproc_worker_utils.py:240]   File "/usr/local/lib/python3.12/dist-packages/vllm/model_executor/layers/linear.py", line 1143, in forward
(VllmWorkerProcess pid=350) ERROR 01-29 20:00:47 multiproc_worker_utils.py:240]     output = tensor_model_parallel_all_reduce(output_parallel)
(VllmWorkerProcess pid=350) ERROR 01-29 20:00:47 multiproc_worker_utils.py:240]              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(VllmWorkerProcess pid=350) ERROR 01-29 20:00:47 multiproc_worker_utils.py:240]   File "/usr/local/lib/python3.12/dist-packages/vllm/distributed/communication_op.py", line 11, in tensor_model_parallel_all_reduce
(VllmWorkerProcess pid=350) ERROR 01-29 20:00:47 multiproc_worker_utils.py:240]     return get_tp_group().all_reduce(input_)
(VllmWorkerProcess pid=350) ERROR 01-29 20:00:47 multiproc_worker_utils.py:240]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(VllmWorkerProcess pid=350) ERROR 01-29 20:00:47 multiproc_worker_utils.py:240]   File "/usr/local/lib/python3.12/dist-packages/vllm/distributed/parallel_state.py", line 347, in all_reduce
(VllmWorkerProcess pid=350) ERROR 01-29 20:00:47 multiproc_worker_utils.py:240]     return torch.ops.vllm.all_reduce(input_, group_name=self.unique_name)
(VllmWorkerProcess pid=350) ERROR 01-29 20:00:47 multiproc_worker_utils.py:240]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(VllmWorkerProcess pid=350) ERROR 01-29 20:00:47 multiproc_worker_utils.py:240]   File "/usr/local/lib/python3.12/dist-packages/torch/_ops.py", line 1116, in __call__
(VllmWorkerProcess pid=350) ERROR 01-29 20:00:47 multiproc_worker_utils.py:240]     return self._op(*args, **(kwargs or {}))
(VllmWorkerProcess pid=350) ERROR 01-29 20:00:47 multiproc_worker_utils.py:240]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(VllmWorkerProcess pid=350) ERROR 01-29 20:00:47 multiproc_worker_utils.py:240]   File "/usr/local/lib/python3.12/dist-packages/vllm/distributed/parallel_state.py", line 109, in all_reduce
(VllmWorkerProcess pid=350) ERROR 01-29 20:00:47 multiproc_worker_utils.py:240]     return group._all_reduce_out_place(tensor)
(VllmWorkerProcess pid=350) ERROR 01-29 20:00:47 multiproc_worker_utils.py:240]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(VllmWorkerProcess pid=350) ERROR 01-29 20:00:47 multiproc_worker_utils.py:240]   File "/usr/local/lib/python3.12/dist-packages/vllm/distributed/parallel_state.py", line 360, in _all_reduce_out_place
(VllmWorkerProcess pid=350) ERROR 01-29 20:00:47 multiproc_worker_utils.py:240]     out = pynccl_comm.all_reduce(input_)
(VllmWorkerProcess pid=350) ERROR 01-29 20:00:47 multiproc_worker_utils.py:240]           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(VllmWorkerProcess pid=350) ERROR 01-29 20:00:47 multiproc_worker_utils.py:240]   File "/usr/local/lib/python3.12/dist-packages/vllm/distributed/device_communicators/pynccl.py", line 124, in all_reduce
(VllmWorkerProcess pid=350) ERROR 01-29 20:00:47 multiproc_worker_utils.py:240]     self.nccl.ncclAllReduce(buffer_type(in_tensor.data_ptr()),
(VllmWorkerProcess pid=350) ERROR 01-29 20:00:47 multiproc_worker_utils.py:240]   File "/usr/local/lib/python3.12/dist-packages/vllm/distributed/device_communicators/pynccl_wrapper.py", line 288, in ncclAllReduce
(VllmWorkerProcess pid=350) ERROR 01-29 20:00:47 multiproc_worker_utils.py:240]     self.NCCL_CHECK(self._funcs["ncclAllReduce"](sendbuff, recvbuff, count,
(VllmWorkerProcess pid=350) ERROR 01-29 20:00:47 multiproc_worker_utils.py:240]   File "/usr/local/lib/python3.12/dist-packages/vllm/distributed/device_communicators/pynccl_wrapper.py", line 254, in NCCL_CHECK
(VllmWorkerProcess pid=350) ERROR 01-29 20:00:47 multiproc_worker_utils.py:240]     raise RuntimeError(f"NCCL error: {error_str}")
(VllmWorkerProcess pid=350) ERROR 01-29 20:00:47 multiproc_worker_utils.py:240] RuntimeError: NCCL error: unhandled cuda error (run with NCCL_DEBUG=INFO for details)
(VllmWorkerProcess pid=350) ERROR 01-29 20:00:47 multiproc_worker_utils.py:240] 
(VllmWorkerProcess pid=350) ERROR 01-29 20:00:47 multiproc_worker_utils.py:240] The above exception was the direct cause of the following exception:
(VllmWorkerProcess pid=350) ERROR 01-29 20:00:47 multiproc_worker_utils.py:240] 
(VllmWorkerProcess pid=350) ERROR 01-29 20:00:47 multiproc_worker_utils.py:240] Traceback (most recent call last):
(VllmWorkerProcess pid=350) ERROR 01-29 20:00:47 multiproc_worker_utils.py:240]   File "/usr/local/lib/python3.12/dist-packages/vllm/executor/multiproc_worker_utils.py", line 234, in _run_worker_process
(VllmWorkerProcess pid=350) ERROR 01-29 20:00:47 multiproc_worker_utils.py:240]     output = run_method(worker, method, args, kwargs)
(VllmWorkerProcess pid=350) ERROR 01-29 20:00:47 multiproc_worker_utils.py:240]              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(VllmWorkerProcess pid=350) ERROR 01-29 20:00:47 multiproc_worker_utils.py:240]   File "/usr/local/lib/python3.12/dist-packages/vllm/utils.py", line 2208, in run_method
(VllmWorkerProcess pid=350) ERROR 01-29 20:00:47 multiproc_worker_utils.py:240]     return func(*args, **kwargs)
(VllmWorkerProcess pid=350) ERROR 01-29 20:00:47 multiproc_worker_utils.py:240]            ^^^^^^^^^^^^^^^^^^^^^
(VllmWorkerProcess pid=350) ERROR 01-29 20:00:47 multiproc_worker_utils.py:240]   File "/usr/local/lib/python3.12/dist-packages/vllm/worker/worker_base.py", line 90, in start_worker_execution_loop
(VllmWorkerProcess pid=350) ERROR 01-29 20:00:47 multiproc_worker_utils.py:240]     output = self.execute_model(execute_model_req=None)
(VllmWorkerProcess pid=350) ERROR 01-29 20:00:47 multiproc_worker_utils.py:240]              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(VllmWorkerProcess pid=350) ERROR 01-29 20:00:47 multiproc_worker_utils.py:240]   File "/usr/local/lib/python3.12/dist-packages/vllm/worker/worker_base.py", line 410, in execute_model
(VllmWorkerProcess pid=350) ERROR 01-29 20:00:47 multiproc_worker_utils.py:240]     output = self.model_runner.execute_model(
(VllmWorkerProcess pid=350) ERROR 01-29 20:00:47 multiproc_worker_utils.py:240]              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(VllmWorkerProcess pid=350) ERROR 01-29 20:00:47 multiproc_worker_utils.py:240]   File "/usr/local/lib/python3.12/dist-packages/torch/utils/_contextlib.py", line 116, in decorate_context
(VllmWorkerProcess pid=350) ERROR 01-29 20:00:47 multiproc_worker_utils.py:240]     return func(*args, **kwargs)
(VllmWorkerProcess pid=350) ERROR 01-29 20:00:47 multiproc_worker_utils.py:240]            ^^^^^^^^^^^^^^^^^^^^^
(VllmWorkerProcess pid=350) ERROR 01-29 20:00:47 multiproc_worker_utils.py:240]   File "/usr/local/lib/python3.12/dist-packages/vllm/worker/multi_step_model_runner.py", line 538, in execute_model
(VllmWorkerProcess pid=350) ERROR 01-29 20:00:47 multiproc_worker_utils.py:240]     output = self._base_model_runner.execute_model(frozen_model_input,
(VllmWorkerProcess pid=350) ERROR 01-29 20:00:47 multiproc_worker_utils.py:240]              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(VllmWorkerProcess pid=350) ERROR 01-29 20:00:47 multiproc_worker_utils.py:240]   File "/usr/local/lib/python3.12/dist-packages/torch/utils/_contextlib.py", line 116, in decorate_context
(VllmWorkerProcess pid=350) ERROR 01-29 20:00:47 multiproc_worker_utils.py:240]     return func(*args, **kwargs)
(VllmWorkerProcess pid=350) ERROR 01-29 20:00:47 multiproc_worker_utils.py:240]            ^^^^^^^^^^^^^^^^^^^^^
(VllmWorkerProcess pid=350) ERROR 01-29 20:00:47 multiproc_worker_utils.py:240]   File "/usr/local/lib/python3.12/dist-packages/vllm/worker/model_runner_base.py", line 146, in _wrapper
(VllmWorkerProcess pid=350) ERROR 01-29 20:00:47 multiproc_worker_utils.py:240]     raise type(err)(f"Error in model execution: "
(VllmWorkerProcess pid=350) ERROR 01-29 20:00:47 multiproc_worker_utils.py:240] RuntimeError: Error in model execution: NCCL error: unhandled cuda error (run with NCCL_DEBUG=INFO for details)
(VllmWorkerProcess pid=351) INFO 01-29 20:00:47 model_runner_base.py:120] Writing input of failed execution to /tmp/err_execute_model_input_20250129-200047.pkl...
(VllmWorkerProcess pid=351) WARNING 01-29 20:00:47 model_runner_base.py:143] Failed to pickle inputs of failed execution: CUDA error: an illegal memory access was encountered
(VllmWorkerProcess pid=351) WARNING 01-29 20:00:47 model_runner_base.py:143] CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.
(VllmWorkerProcess pid=351) WARNING 01-29 20:00:47 model_runner_base.py:143] For debugging consider passing CUDA_LAUNCH_BLOCKING=1
(VllmWorkerProcess pid=351) WARNING 01-29 20:00:47 model_runner_base.py:143] Compile with `TORCH_USE_CUDA_DSA` to enable device-side assertions.
(VllmWorkerProcess pid=351) WARNING 01-29 20:00:47 model_runner_base.py:143] 
[rank3]:[E129 20:00:47.374645555 ProcessGroupNCCL.cpp:1595] [PG ID 2 PG GUID 3 Rank 3] Process group watchdog thread terminated with exception: CUDA error: an illegal memory access was encountered
CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.
For debugging consider passing CUDA_LAUNCH_BLOCKING=1
Compile with `TORCH_USE_CUDA_DSA` to enable device-side assertions.

Exception raised from c10_cuda_check_implementation at ../c10/cuda/CUDAException.cpp:43 (most recent call first):
frame #0: c10::Error::Error(c10::SourceLocation, std::string) + 0x96 (0x7f31b24b9446 in /usr/local/lib/python3.12/dist-packages/torch/lib/libc10.so)
frame #1: c10::detail::torchCheckFail(char const*, char const*, unsigned int, std::string const&) + 0x64 (0x7f31b24636e4 in /usr/local/lib/python3.12/dist-packages/torch/lib/libc10.so)
frame #2: c10::cuda::c10_cuda_check_implementation(int, char const*, char const*, int, bool) + 0x118 (0x7f31b25a5a18 in /usr/local/lib/python3.12/dist-packages/torch/lib/libc10_cuda.so)
frame #3: c10d::ProcessGroupNCCL::WorkNCCL::finishedGPUExecutionInternal() const + 0x56 (0x7f316841e726 in /usr/local/lib/python3.12/dist-packages/torch/lib/libtorch_cuda.so)
frame #4: c10d::ProcessGroupNCCL::WorkNCCL::isCompleted() + 0xa0 (0x7f31684233f0 in /usr/local/lib/python3.12/dist-packages/torch/lib/libtorch_cuda.so)
frame #5: c10d::ProcessGroupNCCL::watchdogHandler() + 0x1da (0x7f316842ab5a in /usr/local/lib/python3.12/dist-packages/torch/lib/libtorch_cuda.so)
frame #6: c10d::ProcessGroupNCCL::ncclCommWatchdog() + 0x14d (0x7f316842c61d in /usr/local/lib/python3.12/dist-packages/torch/lib/libtorch_cuda.so)
frame #7: <unknown function> + 0x145c0 (0x7f31b294e5c0 in /usr/local/lib/python3.12/dist-packages/torch/lib/libtorch.so)
frame #8: <unknown function> + 0x94ac3 (0x7f31b3193ac3 in /usr/lib/x86_64-linux-gnu/libc.so.6)
frame #9: clone + 0x44 (0x7f31b3224a04 in /usr/lib/x86_64-linux-gnu/libc.so.6)

terminate called after throwing an instance of 'c10::DistBackendError'
  what():  [PG ID 2 PG GUID 3 Rank 3] Process group watchdog thread terminated with exception: CUDA error: an illegal memory access was encountered
CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.
For debugging consider passing CUDA_LAUNCH_BLOCKING=1
Compile with `TORCH_USE_CUDA_DSA` to enable device-side assertions.

Exception raised from c10_cuda_check_implementation at ../c10/cuda/CUDAException.cpp:43 (most recent call first):
frame #0: c10::Error::Error(c10::SourceLocation, std::string) + 0x96 (0x7f31b24b9446 in /usr/local/lib/python3.12/dist-packages/torch/lib/libc10.so)
frame #1: c10::detail::torchCheckFail(char const*, char const*, unsigned int, std::string const&) + 0x64 (0x7f31b24636e4 in /usr/local/lib/python3.12/dist-packages/torch/lib/libc10.so)
frame #2: c10::cuda::c10_cuda_check_implementation(int, char const*, char const*, int, bool) + 0x118 (0x7f31b25a5a18 in /usr/local/lib/python3.12/dist-packages/torch/lib/libc10_cuda.so)
frame #3: c10d::ProcessGroupNCCL::WorkNCCL::finishedGPUExecutionInternal() const + 0x56 (0x7f316841e726 in /usr/local/lib/python3.12/dist-packages/torch/lib/libtorch_cuda.so)
frame #4: c10d::ProcessGroupNCCL::WorkNCCL::isCompleted() + 0xa0 (0x7f31684233f0 in /usr/local/lib/python3.12/dist-packages/torch/lib/libtorch_cuda.so)
frame #5: c10d::ProcessGroupNCCL::watchdogHandler() + 0x1da (0x7f316842ab5a in /usr/local/lib/python3.12/dist-packages/torch/lib/libtorch_cuda.so)
frame #6: c10d::ProcessGroupNCCL::ncclCommWatchdog() + 0x14d (0x7f316842c61d in /usr/local/lib/python3.12/dist-packages/torch/lib/libtorch_cuda.so)
frame #7: <unknown function> + 0x145c0 (0x7f31b294e5c0 in /usr/local/lib/python3.12/dist-packages/torch/lib/libtorch.so)
frame #8: <unknown function> + 0x94ac3 (0x7f31b3193ac3 in /usr/lib/x86_64-linux-gnu/libc.so.6)
frame #9: clone + 0x44 (0x7f31b3224a04 in /usr/lib/x86_64-linux-gnu/libc.so.6)

Exception raised from ncclCommWatchdog at ../torch/csrc/distributed/c10d/ProcessGroupNCCL.cpp:1601 (most recent call first):
frame #0: c10::Error::Error(c10::SourceLocation, std::string) + 0x96 (0x7f31b24b9446 in /usr/local/lib/python3.12/dist-packages/torch/lib/libc10.so)
frame #1: <unknown function> + 0xe4271b (0x7f316809971b in /usr/local/lib/python3.12/dist-packages/torch/lib/libtorch_cuda.so)
frame #2: <unknown function> + 0x145c0 (0x7f31b294e5c0 in /usr/local/lib/python3.12/dist-packages/torch/lib/libtorch.so)
frame #3: <unknown function> + 0x94ac3 (0x7f31b3193ac3 in /usr/lib/x86_64-linux-gnu/libc.so.6)
frame #4: clone + 0x44 (0x7f31b3224a04 in /usr/lib/x86_64-linux-gnu/libc.so.6)

CRITICAL 01-29 20:00:47 launcher.py:99] MQLLMEngine is already dead, terminating server process
INFO:     172.19.0.4:53840 - "POST /v1/chat/completions HTTP/1.1" 500 Internal Server Error
INFO:     Shutting down
INFO:     Waiting for application shutdown.
INFO:     Application shutdown complete.
INFO:     Finished server process [7]

🐛 Describe the bug

This is a follow-up to this issue #6042 which was closed because of inactivity. The issue still exists as of v0.7.0 release and we would be able to provide stack-traces as initially requested in the issue (I pasted one in the "Model Input Dump" section of the issue; unfortunately the pickling of problematic inputs fails and the files produced are all empty).

The issue is hard to reproduce because it tends to happen after a while (sometimes a few hours, after vLLM has been running fine).

The issue happens with the following launch config:

      --host=0.0.0.0
      --port=8080
      --model=Llama-3.3-70B-Instruct-FP8-Dynamic
      --disable-log-requests
      --dtype=auto
      --gpu-memory-utilization=0.92
      --kv-cache-dtype=auto
      --load-format=safetensors
      --max-num-seqs=32
      --num_scheduler_steps=16
      --seed=0
      --tensor-parallel-size=4
      --tokenizer-mode=auto

On Docker image: vllm/vllm-openai:v0.7.0@sha256:a43963ed149a7b8b6c8c9dd028d4ab2be9fe804761d41b11cc07043a1edb61a8
Model is Llama-3.3-70B-Instruct quantized to dynamic FP8 (as produced by this script: https://github.com/vllm-project/llm-compressor/blob/main/examples/quantization_w8a8_fp8/llama3_example.py)

We have other models running with no tensor parallelism and they do not seem to exhibit this issue (it is unclear if any tensor parallelism has the issue, or specifically 4).

Before submitting a new issue...

  • Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the documentation page, which can answer lots of frequently asked questions.
@remusao remusao added the bug Something isn't working label Jan 31, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug Something isn't working
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@remusao