cm/psm2 fails to get key from PMIx

[schiotz]

Background information

We are running a cluster with OmniPath communication, using OpenMPI as built by EasyBuild. Upgrading to the latest EasyBuild toolchains means upgrading to OpenMPI 5.X but then cm/psm2 fails with the error Error obtaining unique transport key from PMIX (OMPI_MCA_orte_precondition_transports not present in the environment).. Switching to the ucx PML works, but at a performance cost.

This problem is likely related to #12886, but we already have the SLURM version that @bedroge reports solved the problem for them.

What version of Open MPI are you using? (e.g., v4.1.6, v5.0.1, git branch name and hash, etc.)

We mainly see the problem with OpenMPI version 5.0.8 and PMIx version 5.0.8 (openpmix) [EasyBuild toolchain gompi/2025b], where the problem happens on all jobs started with mpirun, with srun or as singletons.

We see it to a lesser degree with OpenMPI version 5.0.7 and PMIx 5.0.6 [EasyBuild toolchain gompi/2025a], where jobs started with mpirun work, but jobs started with srun or as singletons fail.

Describe how Open MPI was installed (e.g., from a source/distribution tarball, from a git clone, from an operating system distribution package, etc.)

With the EasyBuild build system, i.e. from source tarballs.

If you are building/installing from a git clone, please copy-n-paste the output from git submodule status.

Please describe the system on which you are running

• Operating system/version: RockyLinux 8
• Computer hardware: Dell R650 (69 icelake cores), Dell R640 (40 skylake cores), Huawei XH620 (24 broadwell cores) - same issue on them all
• Network type: OmniPath

---

Details of the problem

All MPI programs crash upon starting. Per default OpenMPI chooses the cm PML with the psm2 MTL.

$ mpirun -np 2 ./mpiBench_2025b
--------------------------------------------------------------------------
Error obtaining unique transport key from PMIX (OMPI_MCA_orte_precondition_transports not present in
the environment).

  Local host: a018
--------------------------------------------------------------------------
--------------------------------------------------------------------------
It looks like MPI_INIT failed for some reason; your parallel process is
likely to abort.  There are many reasons that a parallel process can
fail during MPI_INIT; some of which are due to configuration or environment
problems.  This failure appears to be an internal failure; here's some
additional information (which may only be relevant to an Open MPI
developer):

  PML add procs failed
  --> Returned "Error" (-1) instead of "Success" (0)
--------------------------------------------------------------------------
[a018:00000] *** An error occurred in MPI_Init
[a018:00000] *** reported by process [4230545409,1]
[a018:00000] *** on a NULL communicator
[a018:00000] *** Unknown error
[a018:00000] *** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
[a018:00000] ***    and MPI will try to terminate your MPI job as well)
--------------------------------------------------------------------------
prterun has exited due to process rank 0 with PID 0 on node a018 calling
"abort". This may have caused other processes in the application to be
terminated by signals sent by prterun (as reported here).
--------------------------------------------------------------------------

Note that setting OMPI_MCA_pml=ucx solves the problem, but with OpenMPI 5.0.7 (where we can run with mpirun but not srun or singletons) we measure a 25% performance degradation doing that.

[rhc54]

The problem is simply that the vendor hasn't been supporting/maintaining the OmniPath component since that vendor was spun out of Intel - and Intel stopped supporting it when that happened. Adding to the problem is that Intel subsequently reused the PCI ID that had been assigned to the OPA device, and the new vendor failed to register a replacement. Combined, we were seeing the OPA support being activated where no OmniPath device was present, and bitrot (since I have no ability to test it) resulted in it erroneously throwing errors.

The PMIx component was therefore removed in PMIx v5.0.8. I have no objection to restoring a properly fixed and maintained version of it, but someone (either the vendor or someone else) would have to step up to do those things. In the interim, just set that envar to some arbitrary string of two 64 bit numbers printed in hex with a dash between and zero padding - something like (untested):

OMPI_MCA_orte_precondition_transports=012346789abcdef-012346789abcdef

[schiotz]

Thank you very much for your quick reply. Will the numbers have to be unique in each slurm job, or could it be the same globally on the cluster?
EDIT: I see in the EasyBuild slack discussion that it needs to be unique for each job.

Also, you write that the PMIx component was removed in v5.0.8, but we see the problem at least partially in OpenMPI version 5.0.7 and PMIx 5.0.6, where e.g. singleton jobs crash with the same error.

[schiotz]

Thank you very much @rhc54 !

I can confirm that setting OMPI_MCA_orte_precondition_transports fixes the problem. We will add setting that to a random value in a SLURM prolog script.

[rhc54]

Singletons don't call down into PMIx to get things like network keys - nobody to communicate with, really. OMPI's MPI layer should be smart enough not to error out in that scenario. If it is, then that's a bug.

Note that new keys will be set for child jobs started by comm_spawn, so communicating between parent and child jobs requires that the fabric deal with multiple keys. Not all do.

[schiotz]

Singletons don't call down into PMIx to get things like network keys

That is also what I would expect. But still, the psm2 layer crashes in that scenario, and setting OMPI_MCA_orte_precondition_transports fixes it.

It now looks like the problems I see in OpenMPI 5.0.7 and 5.0.8 have different cause, although the error message from psm2 is the same. The problem in 5.0.8 (where all jobs are affected) are undoubtedly due to the issue you described above. The problem in 5.0.7 (where mpirun works, but srun and singletons fail) is possibly related to our SLURM being built without PMIx support - that will be fixed in the coming week.

[ohlmann]

I also stumbled across this issue. The value given above for OMPI_MCA_orte_precondition_transports is too short, I needed to set it to something like OMPI_MCA_orte_precondition_transports=13241234acffedeb-abcdefabcdef1233, i.e., with 16 places before and after the dash.

The relevant commit in the slurm code base is SchedMD/slurm@c3e63ff