[QUESTION] Gloo connectFullMesh failed when the number of nodes setting "export GLOO_SOCKET_IFNAME=bond4" exceeds 60

[Genlovy-Hoo]

When we train models on a multi-node cluster, it will raise "RuntimeError: Gloo connectFullMesh failed ..." if the number of nodes setting "export GLOO_SOCKET_IFNAME=bond4" exceeds 60, such as 64. And it works when the "bond4-nodes" is less than or equal to 60.

Are there any restrictions for using the Gloo backend with the bond4 network configuration during training?

[Advait251206]

This is a good observation, and it’s not really a Megatron-LM limitation—it comes from Gloo’s connection model + network interface behavior, not the model code itself.

---

1. What GLOO connectFullMesh failed actually means

When you see:

RuntimeError: Gloo connectFullMesh failed

it means:

Gloo failed to establish a full pairwise TCP connection between all ranks.

Gloo uses a full mesh topology, meaning:

every node connects to every other node

So with N nodes, you get:

N × (N - 1) / 2 connections

---

2. Why it breaks around ~60 nodes

When you scale to ~60+ nodes on a single interface (bond4), you hit practical system limits:

(A) File descriptor limit (most common)

Each connection = 1 socket.

At 64 nodes:

64 × 63 / 2 = 2016 TCP connections

That’s per process group, often multiplied by:

• data parallel workers
• tensor parallel groups
• pipeline groups

So real socket count becomes much larger.

Many systems hit:

ulimit -n (max open files)

Typical default is:

• 1024
• 4096 (still too low)

---

(B) Gloo is not designed for large-scale all-to-all control traffic

Gloo is primarily used for:

• initialization (rank discovery)
• parameter broadcast
• small collectives fallback

It is NOT optimized for:

• large node counts
• high fan-out topology
• heavy communication

---

(C) bond4 interface limitation / routing behavior

Using:

export GLOO_SOCKET_IFNAME=bond4

means:

• all Gloo TCP connections are forced through a single bonded interface
• bond interfaces may have:
  • connection tracking limits (conntrack)
  • ARP table limits
  • NIC driver constraints
  • switch-level limits

At scale (~60+ nodes), these become visible.

---

3. Why it works below 60 nodes

Below that threshold:

• socket count is still manageable
• ARP + routing tables are stable
• connection establishment completes before timeouts

So it looks like a “magic number”, but it’s actually system-dependent.

---

4. Is this a Megatron-LM issue?

No.

Megatron-LM only calls:

torch.distributed.init_process_group(backend="gloo")

The failure happens inside:

PyTorch distributed → Gloo → TCP store → full mesh handshake

---

5. Recommended fixes

Fix 1: Don’t use Gloo for large clusters

Use:

export NCCL_SOCKET_IFNAME=bond4

and ensure initialization uses:

backend = nccl

Gloo should only be fallback.

---

Fix 2: Force smaller Gloo role (best practice)

If Gloo is required for init:

export GLOO_SOCKET_IFNAME=lo

or a management network instead of bond4.

This reduces contention.

---

Fix 3: Increase OS limits

On all nodes:

ulimit -n 65536

and ensure:

net.core.somaxconn
net.ipv4.ip_local_port_range

are tuned.

---

Fix 4: Avoid full-mesh explosion (advanced)

PyTorch has:

TORCH_DISTRIBUTED_GLOO_SOCKET_IFNAME
TORCH_DISTRIBUTED_DEBUG

but fundamentally:

Gloo is not designed for 60+ node full-mesh setups

---

6. Best practice in Megatron-style training

For large-scale training:

• NCCL → all heavy comms (TP/PP/DP)
• Gloo → minimal initialization only
• avoid using Gloo for actual training communication

---

Final conclusion

✔ The failure is due to Gloo full-mesh scaling limits, not Megatron-LM
✔ ~60 nodes is where socket + bond interface constraints typically break
✔ bond4 makes it worse due to aggregation complexity
✔ real fix is: use NCCL instead of Gloo for large clusters

---

If you want, I can also explain how Megatron-LM splits process groups (TP/DP/PP) and why only some of them ever use Gloo during startup—that part makes this issue much clearer.

---

If this answer helped or pointed you in the right direction, I'd appreciate it if you could mark it as the accepted answer so it's easier for others with the same issue to find.

Also, if you found my contribution useful, I'd appreciate it if you could check out my GitHub profile, follow me, and star any repositories you find interesting.

GitHub: https://github.com/Advait251206