some question about CUDA_VISIBLE_DEVICES

[wangfakang]

Why is the value of CUDA_VISIBLE_DEVICES not configured in ascending order? For example, CUDA_VISIBLE-DEVICES=0,1,2,3,4,5,6,7 better suited for PXN?

training_results_v3.1/Azure+NVIDIA/benchmarks/gpt3/implementations/pytorch/config_common.sh

Line 1 in 5b62935

export CUDA_VISIBLE_DEVICES=0,4,2,6,1,5,3,7

[wangfakang]

friendly ping @nathanw-mlc @nv-rborkar

[erhoo82]

We used CUDA_VISIBLE_DEVICES=0,4,2,6,1,5,3,7 to work around a bug in NCCL that causes NIC port usage conflict at a specific tensor-parallel size. We fixed the bug and using the ascending order mapping should yield the same performance.

[wangfakang]

We used CUDA_VISIBLE_DEVICES=0,4,2,6,1,5,3,7 to work around a bug in NCCL that causes NIC port usage conflict at a specific tensor-parallel size. We fixed the bug and using the ascending order mapping should yield the same performance.

@erhoo82 Thank you for your reply. Can you explain why used CUDA_VISIBLE_DEVICES=0,4,2,6,1,5,3,7 can work around it？ or is there a PR related to NCCL repair? Thank you.

[wangfakang]

@pgmpablo157321 @hiwotadese @nv-rborkar @erhoo82 have any updates? and another question that why we need to disable NVLS and CUMEM feature ?

training_results_v3.1/Azure+NVIDIA/benchmarks/gpt3/implementations/pytorch/config_common.sh

Lines 18 to 19 in 5b62935

	export NCCL_CUMEM_ENABLE=0
	export NCCL_NVLS_ENABLE=0