R2CCL is a fault tolerant communication library that provides lossless, low overhead failover by exploiting multi-NIC hardware. It is designed as a drop in replacement for NCCL to minimize full job terminations from network failures.
📢 Update 02/23/2026: Added a CloudLab image and public profile for quick reproduction — see Demo.
Megatron Training Performance Evaluation
🔥 Zero-Downtime Hot Repair: Automatically detects and mitigates network failures mid-collective. By utilizing multi-NIC GPU buffer registration and DMA-buffer rollback, R2CC live-migrates failed connections to backup links without losing in-flight data.
⚖️ Topology-Aware Load Balancing (R2CC-Balance): After a failure, R2CC dynamically redistributes traffic across the remaining healthy NICs. It is fully aware of PCIe, NUMA, and NVLink (PXN) topology to maximize remaining bandwidth.
🚀 Failure-Optimized AllReduce (R2CC-AllReduce): Introduces a novel schedule that prevents degraded servers from bottlenecking the cluster by intelligently combining global and partial AllReduce operations. The current implementation builds the schedule from NCCL collectives on sub-communicators: an AllReduce of the first (1−X) of the data on all ranks, a partial AllReduce of the remaining X on the healthy servers, and a pipelined Reduce-to-helper + Broadcast of that tail back to the degraded server (X = lost bandwidth fraction).
R2CC_Demo_with_caption.mov
We provide a pre-built CloudLab image, the test scripts and the logs of running them, so the demo and the other experiments can be reproduced on two r7525 servers:
- Setup guide — instantiate the profile with the pre-built image, flash the SmartNICs, dump the topology.
- Experiments — six ready-to-run tests.
- Logs — the terminal output of running these experiments on CloudLab, one run per test.
- Live Migration: Seamless failover via multi-NIC registration and DMA rollback. ✔️
- R2CCL-Balance: Load-balancing for remaining healthy interfaces. ✔️
- R2CCL-AllReduce: Correct implementation on top of NCCL sub-communicators (partial AllReduce on the healthy servers + pipelined Reduce/Broadcast of the tail), verified with nccl-tests
-c 1and real NIC disconnects. ✔️ - CloudLab r7525 examples and test scripts. ✔️
- Native implementation of R2CCL-AllReduce with a customized kernel (single-pass Stage 2).
We provide a complete set of test scripts and the results of running them in examples/cloudlab_r7525: hot repair of a real NIC failure (injected at runtime with an OVS drop rule on the BlueField SmartNIC) followed by R2CCL-Balance or R2CCL-AllReduce, and nccl-tests correctness/bandwidth runs of plain NCCL, Balance and R2CCL-AllReduce. Each script saves its terminal output; the saved run of every test is in examples/cloudlab_r7525/logs. See examples/cloudlab_r7525/README.md for the scripts, the annotated results and the testbed caveats, and r7525_setup.md for setting up the two CloudLab servers.
git clone https://github.com/r2cc-project/R-2CCL.git
cd R-2CCL
make -jSimilar to NCCL, R2CCL can be benchmarked using nccl-tests. Below we provide compilation commands for nccl-tests and an example of performance testing using allreduce.
git clone https://github.com/NVIDIA/nccl-tests.git
cd nccl-tests
make MPI=1 MPI_HOME=<openmpi> NCCL_HOME=<R-2CCL>/build
mpirun -np 4 -host A,B ./build/all_reduce_perf -b 8K -e 8G -f 2 -t 1 -g 1 -c 1To simplify testing the performance and reduce the complexity of triggering failures (e.g., using SmartNICs to disable specific routing at runtime), we provide environment variables to directly simulate specific scenarios and measure performance. All variables must be identical on every rank (pass them with mpirun -x); the library cross-checks them and warns on mismatches.
R2CC_MODE:
0: NCCL baseline1: Live Migration (static backup connection)2: R2CCL-Balance3: R2CCL-AllReduce
Failure model for modes 2/3 (no real failure is injected):
R2CC_FAILED_HCA=<name|index>[,…]: NIC(s) considered failed (default: the lastR2CC_FAILED_NIC_COUNTNICs).R2CC_FAILED_NODE=<n>: server whose NICs failed (default 0).
R2CC-AllReduce knobs (mode 3): R2CC_AR_STAGE2_CHUNKS (pipeline depth, default 4), R2CC_AR_SCHEDULE (0 all stages concurrent, 1 Stage 2 after Stage 1, 2 also serialize the partial AllReduce — default, 3 also serialize Reduce/Broadcast chunks), R2CC_AR_MIN_BYTES (below this AllReduce falls back to Balance, default 16 MiB).
After a real hot repair: R2CC_AR_AFTER_REPAIR=2|3 selects Balance (default) or R2CC-AllReduce for the collectives that follow the repair.
Failover trigger: NCCL_R2CC_FAILOVER_TIMEOUT_MS (default 5000). A connection whose posted sends show no completion for this long is failed over to its backup path even if the NIC reports no error (a black-holed RoCE path); <= 0 disables the timer and failover then relies on the IB retry timeout alone (NCCL_IB_TIMEOUT x NCCL_IB_RETRY_CNT).
Test Migration Performance. Requirements: 2 nodes, >=2 NICs per node.
# no failure
mpirun -np 4 -host A,B ./build/all_reduce_perf -b 8K -e 8G -f 2 -t 1 -g 1
# 1 failure (static backup connection)
mpirun -x R2CC_MODE=1 -np 4 -host A,B ./build/all_reduce_perf -b 8K -e 8G -f 2 -t 1 -g 1
# or run the first command and disable the routing on the SmartNIC (see the CloudLab example)When only one NIC remains on each node, the performance of different strategies is identical. Therefore, we recommend using machines with 8 NICs and 8 GPUs for testing. Below are the performance tests for R2CCL-Balance and R2CCL-AllReduce, respectively (NIC mlx5_7 of server 0 assumed failed).
mpirun -x R2CC_MODE=2 -x R2CC_FAILED_NODE=0 -x R2CC_FAILED_HCA=mlx5_7 -np 16 -host A,B ./build/all_reduce_perf -b 8K -e 8G -f 2 -t 1 -g 1 -c 1
mpirun -x R2CC_MODE=3 -x R2CC_FAILED_NODE=0 -x R2CC_FAILED_HCA=mlx5_7 -np 16 -host A,B ./build/all_reduce_perf -b 8K -e 8G -f 2 -t 1 -g 1 -c 1@article{wang2025reliable,
title={Reliable and Resilient Collective Communication Library for LLM Training and Serving},
author={Wang, Wei and Yu, Nengneng and Xiong, Sixian and Liu, Zaoxing},
journal={arXiv preprint arXiv:2512.25059},
year={2025}
}