glm52: two NCCL copies in one process corrupt the TP4 prefill comm (missing libnccl.so symlink in shared lib dir)

[xiaguan]

Running the TP4 prefill-only engine bare-metal with LD_LIBRARY_PATH pointing at NCCL 2.30.7 (the shared-filesystem copy the DeepEP decode fleet needs), the launch-time preflight_prefill_kernels dies on every rank:

failed to start GLM5.2 engine
GLM5.2 prefill NCCL all-reduce failed: ncclInvalidArgument

Right after rank models built (Tp4 contexts up) — i.e. ncclCommInitRank succeeds, the first ncclAllReduce (bf16, via cudarc, 4 comms in one process, one per worker thread) is rejected.

The same binary works when the prefill role loads an older NCCL: the P container image ships a 2.28 system NCCL (every prior P validation ran there), and the bare-metal workaround is running P against the host's 2.27.7 (P does not need 2.30.7 — DeepEP symbols are decode-only). Reproduced twice on two different nodes; deterministic.

Suspects to check when root-causing:

• cudarc's ncclAllReduce argument surface vs 2.30's stricter validation (bf16 datatype id, comm/stream binding),
• 2.30's window/registration requirements interacting with plain unregistered device buffers,
• multi-comm-per-process init pattern (4 worker threads, one comm each, same unique id) under 2.30.

Workaround in place: slurm/glm52_2p1d_task.sh keeps the prefill role on the system NCCL and only exports the 2.30.7 path for the decode role. Not release-blocking while P runs containerized or on <=2.28, but it blocks unifying P/D onto one bare-metal library set.

🤖 Generated with Claude Code

Root cause (found same day)

Not an NCCL 2.30.7 incompatibility — a two-NCCL-copies-in-one-process split:

• The shared-filesystem copy only shipped libnccl.so.2 + libnccl.so.2.30.7 — no unversioned libnccl.so.
• cudarc's dynamic loader opens the unversioned name, missed the LD_LIBRARY_PATH dir, and fell through to the system 2.27.7 (NCCL version 2.27.7+cuda13.0 banner in the failing run).
• The TP bootstrap unique id, however, comes from the DeepEP shim (glm52_ep_deepep_unique_id), which links libnccl.so.2 → 2.30.7. A 2.30-format unique id fed into 2.27.7's ncclCommInitRank "succeeds", then the first ncclAllReduce fails CommCheck: NCCL WARN Error: corrupted comm object detected → surfaced as ncclInvalidArgument.
• Containers never hit this because the deb install provides every symlink, so both loaders resolve to the same 2.30.7.

Fix: ln -s libnccl.so.2.30.7 libnccl.so in the shared copy — one library instance again; TP4 prefill comes up bare-metal.

Hardening worth doing (leaving this issue open for it): the engine should fail loud at startup when the unique-id producer and the comm library disagree on version — e.g. log ncclGetVersion from both linkage paths (cudarc + shim) and ensure! they match. A corrupted-comm crash three minutes into weight load is an expensive way to learn a symlink is missing.

[github-actions]

This issue has been inactive for 14 days. It will be closed after another 30 days unless there is new activity.