Shm failure with PSM2

[adrianjhpc]

Running using Intel MPI and PSM2 on a dual rail Omnipath network we're getting these errors with some applications:

Error opening remote shared memory object in shm_open: No such file or directory (err=9)
PSM could not set up shared memory segment (err=9)

When we look in /dev/shm we see psm2_shm.295510000000000020e02 type files, but it is still failing. We've tried cleaning up /dev/shm but it does not seem to help.

We've seen this for PSM2 10.3.46, 11.2.23, 11.2.77, and 11.2.78.

Any idea what's going wrong?

[mwheinz]

Adrian,

You're problem doesn't ring any bells, but I've opened an internal bug report for it. Could you give me a little more info? What version of IFS are you using and on what distro you're using it?

[mwheinz]

Also - which MPI you're using; if you could provide the mpirun command line it would help us understand what might be going on.

[adrianjhpc]

Thanks.

CentOS Linux release 7.5.1804
Intel(R) MPI Library, Version 2019 Update 3 Build 20190214 (id: b645a4a54)
For MPI run stuff:

export FI_PROVIDER=psm2
export PSM2_MULTIRAIL=1
export PSM2_MULTIRAIL_MAP=0:1,1:1
export PSM2_MULTI_EP=1
export PSM2_DEVICES=self,shm,hfi
export OMP_NUM_THREADS=2
mpirun -genvall -n 960 -ppn 48 ...

Can you "remind me" how to get the IFS version?

[mwheinz]

opaconfig -V should do it.

[adrianjhpc]

opaconfig -V reports:

10.8.0.0.204

[mwheinz]

Thanks. Are you using the version of PSM2 that comes packaged with Intel MPI or the upstream version?

[adrianjhpc]

For this it was the PSM2 with Intel MPI but we do have other versions installed on the system.

[mwheinz]

okay.

[mwheinz]

Adrian, thinking about it, We've never tested PSM2 in conjunction with OMP and we don't provide strong protections for using PSM2 in a multi-threaded environment. Does the problem still exist if you set export OMP_NUM_THREADS=1?

[adrianjhpc]

I can check. I should say that we're not doing any MPI from within OpenMP regions, but I'll check nevertheless.

[adrianjhpc]

Using a single OpenMP thread doesn't help I'm afraid. How would you suggest I debug the issue, can build my own PSM2 source and modify the part failing to see what's going wrong?

Here's a stack trace of the current failure (or at least the relevant part):

#0 pthread_spin_lock () at ../nptl/sysdeps/x86_64/pthread_spin_lock.S:24
#1 0x00002acf2dd7ab00 in psmi_amsh_short_request () from /lib64/libpsm2.so.2
#2 0x00002acf2dd79bdf in amsh_ep_connreq_poll () from /lib64/libpsm2.so.2
#3 0x00002acf2dd7bd4b in amsh_ep_connect () from /lib64/libpsm2.so.2
#4 0x00002acf2dd8aed6 in psm2_ep_connect () from /lib64/libpsm2.so.2

[adrianjhpc]

Debugging PSM2 a bit, the error is happening in this function:

psm2_error_t psmi_shm_create(ptl_t *ptl_gen)

It's this bit of code that's failing:

    for (iterator = 0; iterator <= INT_MAX; iterator++) {
            snprintf(shmbuf,
                     sizeof(shmbuf),
                     "/psm2_shm.%ld%016lx%d",
                     (long int) getuid(),
                     epid,
                     iterator);
            dest_shmfd = shm_open(shmbuf, O_RDWR, S_IRWXU);
            if (dest_shmfd < 0) {
                    if (errno == EACCES && iterator < INT_MAX)
                            continue;
                    else {
                            err = psmi_handle_error(NULL,
                                                    PSM2_SHMEM_SEGMENT_ERR,
                                                    "Error opening remote "
                                                    "shared memory object "
                                                    "in shm_open: %s",
                                                    strerror(errno));

                    goto fail;
            }
            shmfd =
                shm_open(amsh_keyname, O_RDWR, S_IRUSR | S_IWUSR);

Where it is looking for a specific psm2 file in /dev/shm that isn't on the current host but is on a remote host (I can find it by searching all the /dev/shm on the hosts that have been used for the run). For instance, this failed on node 22 but the file it failed on (psm2_shm.2955100000000001624020) was on node 17.

[mwheinz]

Okay - try a workaround. in your mpirun line add

-X PSM2_DEVICES=self,hfi

That will disable the shm device. I have no explanation for why a machine would be trying to open a shared memory handle on a different machine.

[adrianjhpc]

It definitely works if we disable the shm device, we're just trying to get shm to work for better performance.

[mwheinz]

It definitely works if we disable the shm device, we're just trying to get shm to work for better performance.

Adrian, I know it's been 10 days, I just wanted to let you know we are looking at this.