MPICH osu_acc_latency, osu_get_acc_latency hang at 16k size

[ztiffany]

jupiter$ LD_LIBRARY_PATH=/cray/css/users/ztiffany/libfab/install/lib/ srun -n 2 -N 2 ldd /cray/css/users/libfabric-test/clients/mpich/osu-micro-benchmarks-5.0/mpi/one-sided/osu_acc_latency
	linux-vdso.so.1 =>  (0x00007ffff7ffe000)
	libpmi.so.0 => /opt/cray/lib64/libpmi.so.0 (0x00007ffff7d58000)
	libmpi.so.12 => /cray/css/users/libfabric-test/clients/mpich/install/lib/libmpi.so.12 (0x00007ffff78be000)
	libc.so.6 => /lib64/libc.so.6 (0x00007ffff7542000)
	libpthread.so.0 => /lib64/libpthread.so.0 (0x00007ffff7325000)
	libdl.so.2 => /lib64/libdl.so.2 (0x00007ffff7120000)
	/lib64/ld-linux-x86-64.so.2 (0x0000555555554000)
	libfabric.so.1 => /cray/css/users/ztiffany/libfab/install/lib/libfabric.so.1 (0x00007ffff6e4c000)
	libwlm_detect.so.0 => /opt/cray/wlm_detect/default/lib64/libwlm_detect.so.0 (0x00007ffff6c49000)
	libalpsutil.so.0 => /opt/cray/alps/default/lib64/libalpsutil.so.0 (0x00007ffff6a46000)
	libalpslli.so.0 => /opt/cray/alps/default/lib64/libalpslli.so.0 (0x00007ffff6841000)
	libugni.so.0 => /opt/cray/ugni/default/lib64/libugni.so.0 (0x00007ffff65cd000)
	librt.so.1 => /lib64/librt.so.1 (0x00007ffff63c4000)
	libgfortran.so.3 => /usr/lib64/libgfortran.so.3 (0x00007ffff60ae000)
	libm.so.6 => /lib64/libm.so.6 (0x00007ffff5e34000)
	libgcc_s.so.1 => /lib64/libgcc_s.so.1 (0x00007ffff5c1e000)
	libquadmath.so.0 => /usr/lib64/libquadmath.so.0 (0x00007ffff59e8000)
	libxpmem.so.0 => /opt/cray/xpmem/default/lib64/libxpmem.so.0 (0x00007ffff57e4000)
	libudreg.so.0 => /opt/cray/udreg/default/lib64/libudreg.so.0 (0x00007ffff55db000)
	linux-vdso.so.1 =>  (0x00007ffff7ffe000)
	libpmi.so.0 => /opt/cray/lib64/libpmi.so.0 (0x00007ffff7d58000)
	libmpi.so.12 => /cray/css/users/libfabric-test/clients/mpich/install/lib/libmpi.so.12 (0x00007ffff78be000)
	libc.so.6 => /lib64/libc.so.6 (0x00007ffff7542000)
	libpthread.so.0 => /lib64/libpthread.so.0 (0x00007ffff7325000)
	libdl.so.2 => /lib64/libdl.so.2 (0x00007ffff7120000)
	/lib64/ld-linux-x86-64.so.2 (0x0000555555554000)
	libfabric.so.1 => /cray/css/users/ztiffany/libfab/install/lib/libfabric.so.1 (0x00007ffff6e4c000)
	libwlm_detect.so.0 => /opt/cray/wlm_detect/default/lib64/libwlm_detect.so.0 (0x00007ffff6c49000)
	libalpsutil.so.0 => /opt/cray/alps/default/lib64/libalpsutil.so.0 (0x00007ffff6a46000)
	libalpslli.so.0 => /opt/cray/alps/default/lib64/libalpslli.so.0 (0x00007ffff6841000)
	libugni.so.0 => /opt/cray/ugni/default/lib64/libugni.so.0 (0x00007ffff65cd000)
	librt.so.1 => /lib64/librt.so.1 (0x00007ffff63c4000)
	libgfortran.so.3 => /usr/lib64/libgfortran.so.3 (0x00007ffff60ae000)
	libm.so.6 => /lib64/libm.so.6 (0x00007ffff5e34000)
	libgcc_s.so.1 => /lib64/libgcc_s.so.1 (0x00007ffff5c1e000)
	libquadmath.so.0 => /usr/lib64/libquadmath.so.0 (0x00007ffff59e8000)
	libxpmem.so.0 => /opt/cray/xpmem/default/lib64/libxpmem.so.0 (0x00007ffff57e4000)
	libudreg.so.0 => /opt/cray/udreg/default/lib64/libudreg.so.0 (0x00007ffff55db000)
jupiter$ LD_LIBRARY_PATH=/cray/css/users/ztiffany/libfab/install/lib/ srun -n 2 -N 2 /cray/css/users/libfabric-test/clients/mpich/osu-micro-benchmarks-5.0/mpi/one-sided/osu_acc_latency
# OSU MPI_Accumulate latency Test v5.0
# Window creation: MPI_Win_allocate
# Synchronization: MPI_Win_flush
# Size          Latency (us)
0                       0.05
1                      12.97
2                      12.91
4                      12.86
8                      12.86
16                     13.02
32                     13.06
64                     13.03
128                    13.14
256                    13.06
512                    13.19
1024                   13.79
2048                   14.69
4096                   16.59
8192                   20.60
(hung)

[sungeunchoi]

Were you running head of master? Can you try with 9a40e91 that excludes Howard's CM NIC fix?

[hppritcha]

Is this MPICH/CH3 or CH4?

[sungeunchoi]

CH3

[hppritcha]

CH3 had bugs in at least one of the MPI RMA operations. I recall helping the Intel MPICH crew to fix this in their IMPI, but I do not know if that was pushed back upstream.

We really should be testing with CH4.

[hppritcha]

it was related to multi iov support. If you disable multi-iov support in gni provider the test should pass - if I remember the bug correctly.