PMIx_Fence_nb result in `shepherd died from signal 11`

[hzhou]

Point of Contact

Hui Zhou

Contact Details

zhouh@anl.gov

Vendor/ALCF/other tickets/IDs

No response

Reproducer Path

None

Status

Open

Details

MPICH uses PMIx_Fence_nb to support MPI Sessions. However, on Aurora this resulted in PALs error: shepherd died from signal 11. This happens even with traditional MPI_Init when PMIx_Fecnce_nb is called with all the processes.

Currently as a work-around, MPICH sets MPIR_CVAR_PMI_DISABLE_GROUP=1, which uses PMIx_Fence(&pmix_wcproc, 1, info, 1); instead.

To reproduce, create a dummy MPI program that includes MPI_Init/MPI_Finalize, set MPIR_CVAR_PMI_DISABLE_GROUP=0, and run with multiple nodes with e.g. ppn 12.

For context, the relevant MPICH codes are (https://github.com/pmodels/mpich/blob/e7b0c5ca4ffd2db03231cfa28b900be9a1220150/src/util/mpir_pmix.inc#L335-L384)

static void fence_cb(pmix_status_t status, void *cbdata)
{
    MPL_atomic_store_int((MPL_atomic_int_t *) cbdata, 1);
}


static int pmix_barrier_group(int *group, int count, const char *stringtag)
{
    int mpi_errno = MPI_SUCCESS;
    int pmi_errno;


    if (group == MPIR_PMI_GROUP_SELF) {
        /* noop */
        goto fn_exit;
    }


    pmix_proc_t *procs = NULL;
    PMIX_PROC_CREATE(procs, count);
    for (int i = 0; i < count; i++) {
        PMIX_LOAD_PROCID(&(procs[i]), pmix_proc.nspace, group[i]);
    }


    pmix_info_t *info;
    int flag = 1;
    PMIX_INFO_CREATE(info, 1);
    PMIx_Info_load(info, PMIX_COLLECT_DATA, &flag, PMIX_BOOL);


    MPL_atomic_int_t done_flag;
    MPL_atomic_store_int(&done_flag, 0);


    pmi_errno = PMIx_Fence_nb(procs, count, info, 1, fence_cb, &done_flag);
    if (pmi_errno == PMIX_OPERATION_SUCCEEDED) {
        MPL_atomic_store_int(&done_flag, 1);
    }
    MPIR_ERR_CHKANDJUMP1(pmi_errno != PMIX_SUCCESS, mpi_errno, MPI_ERR_OTHER, "**pmix_fence_nb",
                         "**pmix_fence_nb %d", pmi_errno);
    while (MPL_atomic_load_int(&done_flag) == 0) {
        /* NOTE: PMIx_Progress not needed unless progress threads are disabled */
#if 0
        PMIx_Progress();
#endif
    }


    PMIX_INFO_FREE(info, 1);
    PMIX_PROC_FREE(procs, count);


  fn_exit:
    return mpi_errno;
  fn_fail:
    goto fn_exit;
}

Is this a priority/blocking bug?

• Priority

ETA

No response