[Issue]: Incompatibility with libfabric 1.22 - RCCL hangs on multinode PyTorch distributed training

[dipietrantonio]

Problem Description

Dear ROCm developers, it seems that this plug in is not compatible with the latest version of lib fabric present on Setonix (Pawsey Supercomputing Centre). I am trying to run a simple distributed training job. on 2 nodes and the execution hangs right at the start. After some time, I get the following error messages:

[rank14]: Traceback (most recent call last):
[rank14]:   File "/software/projects/pawsey0001/cdipietrantonio/machine-learning-examples/pytorch/mnist_ddp/mnist.py", line 100, in <module>
[rank14]:     main()
[rank14]:   File "/software/projects/pawsey0001/cdipietrantonio/machine-learning-examples/pytorch/mnist_ddp/mnist.py", line 97, in main
[rank14]:     train(args.epochs)
[rank14]:   File "/software/projects/pawsey0001/cdipietrantonio/machine-learning-examples/pytorch/mnist_ddp/mnist.py", line 55, in train
[rank14]:     model = DistributedDataParallel(model, device_ids=[local_rank])
[rank14]:             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank14]:   File "/usr/local/lib/python3.12/dist-packages/torch/nn/parallel/distributed.py", line 835, in __init__
[rank14]:     _verify_param_shape_across_processes(self.process_group, parameters)
[rank14]:   File "/usr/local/lib/python3.12/dist-packages/torch/distributed/utils.py", line 282, in _verify_param_shape_across_processes
[rank14]:     return dist._verify_params_across_processes(process_group, tensors, logger)
[rank14]:            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank14]: RuntimeError: DDP expects same model across all ranks, but Rank 14 has 10 params, while rank 2 has inconsistent 0 params.
[rank12]:[E916 13:33:59.240756795 ProcessGroupNCCL.cpp:1746] [PG ID 0 PG GUID 0(default_pg) Rank 12] Received a dump signal due to a collective timeout from this local rank and we will try our best to dump the debug info. Last enqueued NCCL work: 1, last completed NCCL work: -1.This is most likely caused by incorrect usages of collectives, e.g., wrong sizes used across ranks, the order of collectives is not same for all ranks or the scheduled collective, for some reason, didn't run. Additionally, this can be caused by GIL deadlock or other reasons such as network errors or bugs in the communications library (e.g. NCCL), etc. 
[rank12]:[E916 13:33:59.240861047 ProcessGroupNCCL.cpp:1536] [PG ID 0 PG GUID 0(default_pg) Rank 12] ProcessGroupNCCL preparing to dump debug info. Include stack trace: 1
[rank12]:[E916 13:33:59.241590278 FlightRecorder.cpp:162] Error opening file for writing Flight Recorder debug info: /home/cdipietrantonio/.cache/torch/nccl_trace_rank_12
[rank8]:[E916 13:33:59.247535519 ProcessGroupNCCL.cpp:1746] [PG ID 0 PG GUID 0(default_pg) Rank 8] Received a dump signal due to a collective timeout from this local rank and we will try our best to dump the debug info. Last enqueued NCCL work: 1, last completed NCCL work: -1.This is most likely caused by incorrect usages of collectives, e.g., wrong sizes used across ranks, the order of collectives is not same for all ranks or the scheduled collective, for some reason, didn't run. Additionally, this can be caused by GIL deadlock or other reasons such as network errors or bugs in the communications library (e.g. NCCL), etc. 
[rank8]:[E916 13:33:59.247653928 ProcessGroupNCCL.cpp:1536] [PG ID 0 PG GUID 0(default_pg) Rank 8] ProcessGroupNCCL preparing to dump debug info. Include stack trace: 1
[rank8]:[E916 13:33:59.248455149 FlightRecorder.cpp:162] Error opening file for writing Flight Recorder debug info: /home/cdipietrantonio/.cache/torch/nccl_trace_rank_8
[rank11]:[E916 13:33:59.252456777 ProcessGroupNCCL.cpp:1746] [PG ID 0 PG GUID 0(default_pg) Rank 11] Received a dump signal due to a collective timeout from this local rank and we will try our best to dump the debug info. Last enqueued NCCL work: 1, last completed NCCL work: -1.This is most likely caused by incorrect usages of collectives, e.g., wrong sizes used across ranks, the order of collectives is not same for all ranks or the scheduled collective, for some reason, didn't run. Additionally, this can be caused by GIL deadlock or other reasons such as network errors or bugs in the communications library (e.g. NCCL), etc. 
[rank11]:[E916 13:33:59.252550609 ProcessGroupNCCL.cpp:1536] [PG ID 0 PG GUID 0(default_pg) Rank 11] ProcessGroupNCCL preparing to dump debug info. Include stack trace: 1
[rank10]:[E916 13:33:59.252966254 ProcessGroupNCCL.cpp:1746] [PG ID 0 PG GUID 0(default_pg) Rank 10] Received a dump signal due to a collective timeout from this local rank and we will try our best to dump the debug info. Last enqueued NCCL work: 1, last completed NCCL work: -1.This is most likely caused by incorrect usages of collectives, e.g., wrong sizes used across ranks, the order of collectives is not same for all ranks or the scheduled collective, for some reason, didn't run. Additionally, this can be caused by GIL deadlock or other reasons such as network errors or bugs in the communications library (e.g. NCCL), etc. 
[rank10]:[E916 13:33:59.253081066 ProcessGroupNCCL.cpp:1536] [PG ID 0 PG GUID 0(default_pg) Rank 10] ProcessGroupNCCL preparing to dump debug info. Include stack trace: 1
[rank11]:[E916 13:33:59.253232429 FlightRecorder.cpp:162] Error opening file for writing Flight Recorder debug info: /home/cdipietrantonio/.cache/torch/nccl_trace_rank_11
[rank10]:[E916 13:33:59.253833763 FlightRecorder.cpp:162] Error opening file for writing Flight Recorder debug info: /home/cdipietrantonio/.cache/torch/nccl_trace_rank_10

I am using PyTorch 2.7.1 and ROCm 6.3.3. Libfabric is 1.22.

When I use libfabric 1.15, everything works well.

Operating System

SLES15-SP6

CPU

AMD Trento

GPU

AMD MI250X

ROCm Version

6.4.1

ROCm Component

No response

Steps to Reproduce

No response

(Optional for Linux users) Output of /opt/rocm/bin/rocminfo --support

No response

Additional Information

No response

[juliusroeder]

Hi Team,
I have similar issues.

Target:
Lumi

GPU:
AMD MI250X

ROCm Version:
6.2.4

PyTorch:
2.7.1

Libfabric:
e08d3161b8038ebfa50a4eb5bf76132e4eaeb137
(This seems to be the lowest libfabric that also supports the CXI provider.)

CXI Stack:
CASSINI_HEADERS_VERSION=9a8a738a879f007849fbc69be8e3487a4abf0952
CXI_DRIVER_VERSION=c284516fa8d0027b8f84e8ca818e8c6bbe320bd8
LIBCXI_VERSION=0f3609b5e224636abffc7ceffe6f4e9c83244c08

MPICH
4.2.3

RCCL
85eb1f1

I have tried both 17d41cb and 6dae1b0 from aws-ofi-rccl.

I run this in a singularity container. As of right now only ROCm 6.0.2 is available on Lumi, therefore for newer Pytorch versions I need a container with a newer ROCm.

The following works:

• OSU BW benchmarks
• OSU RCCL benchmarks
• rccltests
• single node Pytorch DDP
• Multi-node DDP with RCCL via sockets

The bandwidth of the benchmarks also look good .

The AllGather of the pytorch DDP process seems to hang with the model not being the same across all ranks.

I have attached the complete log as well as the FlightRecord file.

PyTorchDDP_Hang.txt

nccl_trace_rank_0.txt

I am not certain that this is a aws-ofi-rccl issue.
It may also be related to libfabric, libcxi, cxi driver or rccl. However, I am not sure how to continue debugging this behavior.

[alazzaro]

I'm not sure if there is an internal LUMI ticket

[dipietrantonio]

Currently at Pawsey we have implemented a workaround keeping libfabric 1.15 alongside libfabric 1.22. I will investigate the issue further, most likely next week. Hopefully AMD will be able to provide some insight by then.

[alazzaro]

@dipietrantonio have you considered testing https://github.com/HewlettPackard/open-ofi-xccl ?

[juliusroeder]

@dipietrantonio have you considered testing https://github.com/HewlettPackard/open-ofi-xccl ?

Hi @alazzaro ,
i have tried open-ofi-xccl from the commit with ROCm support - before it was merged. I had some other issues with it but didnt dig deeper into it. I'll rebuild the container and give it a shot again and make an issue on the repo if I run into issues.

[alazzaro]

@juliusroeder on LUMI we have tested some SHS and OFI combinations. This is not for the RCCL tests, unfortunately.

Check the Easybuild modules (cc @mszpindler)

• https://github.com/Lumi-supercomputer/LUMI-EasyBuild-contrib/tree/main/easybuild/easyconfigs/l/libcxi
• https://github.com/Lumi-supercomputer/LUMI-EasyBuild-contrib/tree/2b2ddcdc843cf87757faa693dec32b902ebead05/easybuild/easyconfigs/l/libfabric
• Preliminary recipe for HPE's open-ofi-xccl plugin Lumi-supercomputer/LUMI-EasyBuild-contrib#327

I also suggest to check what @sfantao is doing in his own repository for LUMI at https://github.com/sfantao/LUMI-containers/tree/main/RecipesDocker

[juliusroeder]

I managed to solve the hanging issue on my side by unsetting the two environment variables CXI_FORK_SAFE and CXI_FORK_SAFE_HP. Without these two variables my container runs the simple DDP test I have.