ASAN HOST got SEGV on multiple nodes

[jkwack]

Point of Contact

JaeHyuk Kwack, Saumil Patel, Wendy Wu

Contact Details

No response

Vendor/ALCF/other tickets/IDs

CMPLRLLVM-75815

Reproducer Path

/lus/flare/projects/Tools/jkwack-tools-reproducer/JaeHyuk/ASAN_host_multinodes
source/reproducers/tools/asan_multinode/

Status

Open

Details

Wendy Wu reported address sanitizer for host returns SEGV error on 2 nodes with HARVEY application.
The simple GeoSeries benchmark is used to reproduce the issue.

• Device ASAN works fine on a single node and two nodes
• Host ASAN works fine on a single node with one MPI or two MPI ranks
• Host ASAN returns the following error on two nodes with two MPI ranks

mpirun -n 2 --ppn 1 ./Comp_GeoSeries_sycl-mpicxx_DP_ASAN_HOST
AddressSanitizer:DEADLYSIGNAL
=================================================================
==111900==ERROR: AddressSanitizer: SEGV on unknown address 0x0000000898ce (pc 0x0000000898ce bp 0x000000000000 sp 0x7ffc1a1e1f38 T0)
==111900==The signal is caused by a READ memory access.
x4201c3s1b0n0.hsn.cm.aurora.alcf.anl.gov: rank 0 died from signal 11
x4201c3s3b0n0.hsn.cm.aurora.alcf.anl.gov: rank 1 died from signal 15

The following shows what I tried:

jkwack@x4201c3s1b0n0:/lus/flare/projects/Tools/jkwack-tools-reproducer/JaeHyuk/ASAN_host_multinodes> make
mpicxx -fsycl -g -O1 Comp_GeoSeries_sycl.cpp -o Comp_GeoSeries_sycl_mpicxx_DP 
rm -rf *.o *.mod *.dSYM
mpicxx -fsycl -g -O1 -Xarch_host -fsanitize=address  Comp_GeoSeries_sycl.cpp -o Comp_GeoSeries_sycl-mpicxx_DP_ASAN_HOST 
rm -rf *.o *.mod *.dSYM
mpicxx -fsycl -g -O1 -Xarch_device -fsanitize=address Comp_GeoSeries_sycl.cpp -o Comp_GeoSeries_sycl-mpicxx_DP_ASAN_DEVICE 
rm -rf *.o *.mod *.dSYM


jkwack@x4201c3s1b0n0:/lus/flare/projects/Tools/jkwack-tools-reproducer/JaeHyuk/ASAN_host_multinodes> mpirun -n 1 --ppn 1 ./Comp_GeoSeries_sycl-mpicxx_DP_ASAN_DEVICE 
==== DeviceSanitizer: ASAN
	                 Number of MPI process:      1
	                             Precision: double
	  Number of rows/columns of the matrix:   1024
...
====ERROR: DeviceSanitizer: detected memory leaks of Device USM
Direct leak of 8388608 byte(s) at 0xff00180000c00800 allocated from:
 #0 in main /lus/flare/projects/Tools/jkwack-tools-reproducer/JaeHyuk/ASAN_host_multinodes/Comp_GeoSeries_sycl.cpp:121:33
 #1 in __libc_start_main (/lib64/libc.so.6+0x154e5d23924c)
 #2 in _start /home/abuild/rpmbuild/BUILD/glibc-2.31/csu/../sysdeps/x86_64/start.S:120:0


jkwack@x4201c3s1b0n0:/lus/flare/projects/Tools/jkwack-tools-reproducer/JaeHyuk/ASAN_host_multinodes> mpirun -n 2 --ppn 1 ./Comp_GeoSeries_sycl-mpicxx_DP_ASAN_DEVICE 
==== DeviceSanitizer: ASAN
==== DeviceSanitizer: ASAN
	                 Number of MPI process:      2
	                             Precision: double
	  Number of rows/columns of the matrix:   1024
...
====ERROR: DeviceSanitizer: detected memory leaks of Device USM
Direct leak of 8388608 byte(s) at 0xff00180000c00800 allocated from:
 #1 in __libc_start_main (/lib64/libc.so.6+0x14abb7ade24c)
 #2 in _start /home/abuild/rpmbuild/BUILD/glibc-2.31/csu/../sysdeps/x86_64/start.S:120:0

 #0 in main /lus/flare/projects/Tools/jkwack-tools-reproducer/JaeHyuk/ASAN_host_multinodes/Comp_GeoSeries_sycl.cpp:121:33
 #1 in __libc_start_main (/lib64/libc.so.6+0x14908121b24c)
 #2 in _start /home/abuild/rpmbuild/BUILD/glibc-2.31/csu/../sysdeps/x86_64/start.S:120:0


jkwack@x4201c3s1b0n0:/lus/flare/projects/Tools/jkwack-tools-reproducer/JaeHyuk/ASAN_host_multinodes> mpirun -n 1 --ppn 1 ./Comp_GeoSeries_sycl-mpicxx_DP_ASAN_HOST
	                 Number of MPI process:      1
	                             Precision: double
	  Number of rows/columns of the matrix:   1024
	 The highest order of geometric series:     30
	                 Number of repetitions:     10
	             Memory Usage per MPI rank:    16.777216 MB
		Warming up .....
		Main Computations  10 repetitions ......
		0%                     25%                      50%                     75%                     100%
		||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
		              Error_MPI_{Min,Mean,Max}/MPI =   0.0000e+00    0.0000e+00    0.0000e+00
		             GFLOP-rate_{Min,Mean,Max}/MPI =  1031.235771   1031.235771   1031.235771
		                                 Wall Time =     0.000610 sec
		                                 FLOP-rate =  1031.235771 GFLOP/sec

=================================================================
==111738==ERROR: LeakSanitizer: detected memory leaks

Direct leak of 8388608 byte(s) in 1 object(s) allocated from:
    #0 0x0000004e0848 in malloc /netbatch/donb38620_00/runDir/dir/workspace/NIT/xmain-rel/LX/xmainefi2linux_release/ws/icsws/llvm/compiler-rt/lib/asan/asan_malloc_linux.cpp:67:3
    #1 0x000000525b28 in main /lus/flare/projects/Tools/jkwack-tools-reproducer/JaeHyuk/ASAN_host_multinodes/Comp_GeoSeries_sycl.cpp:114:23
    #2 0x1521e68f724c in __libc_start_main (/lib64/libc.so.6+0x3524c) (BuildId: 171a59c1c43a8f7b93c3dff765aae0b675fe10f6)
    #3 0x000000433239 in _start /home/abuild/rpmbuild/BUILD/glibc-2.31/csu/../sysdeps/x86_64/start.S:120
...
SUMMARY: AddressSanitizer: 16804013 byte(s) leaked in 139 allocation(s).
x4201c3s1b0n0.hsn.cm.aurora.alcf.anl.gov: rank 0 exited with code 1


jkwack@x4201c3s1b0n0:/lus/flare/projects/Tools/jkwack-tools-reproducer/JaeHyuk/ASAN_host_multinodes> mpirun -n 2 --ppn 2 ./Comp_GeoSeries_sycl-mpicxx_DP_ASAN_HOST
	                 Number of MPI process:      2
	                             Precision: double
	  Number of rows/columns of the matrix:   1024
	 The highest order of geometric series:     30
	                 Number of repetitions:     10
	             Memory Usage per MPI rank:    16.777216 MB
		Warming up .....
		Main Computations  10 repetitions ......
		0%                     25%                      50%                     75%                     100%
		||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
		              Error_MPI_{Min,Mean,Max}/MPI =   0.0000e+00    0.0000e+00    0.0000e+00
		             GFLOP-rate_{Min,Mean,Max}/MPI =   913.994831    979.064022   1044.133213
		                                 Wall Time =     0.000688 sec
		                                 FLOP-rate =  1827.989662 GFLOP/sec

=================================================================
==111804==ERROR: LeakSanitizer: detected memory leaks
Direct leak of 8388608 byte(s) in 1 object(s) allocated from:
=================================================================
==111805==ERROR: LeakSanitizer: detected memory leaks
Direct leak of 8388608 byte(s) in 1 object(s) allocated from:
...
SUMMARY: AddressSanitizer: 16902461 byte(s) leaked in 142 allocation(s).
x4201c3s1b0n0.hsn.cm.aurora.alcf.anl.gov: rank 0 exited with code 1


jkwack@x4201c3s1b0n0:/lus/flare/projects/Tools/jkwack-tools-reproducer/JaeHyuk/ASAN_host_multinodes> mpirun -n 2 --ppn 1 ./Comp_GeoSeries_sycl-mpicxx_DP_ASAN_HOST
AddressSanitizer:DEADLYSIGNAL
=================================================================
==111900==ERROR: AddressSanitizer: SEGV on unknown address 0x0000000898ce (pc 0x0000000898ce bp 0x000000000000 sp 0x7ffc1a1e1f38 T0)
==111900==The signal is caused by a READ memory access.
x4201c3s1b0n0.hsn.cm.aurora.alcf.anl.gov: rank 0 died from signal 11
x4201c3s3b0n0.hsn.cm.aurora.alcf.anl.gov: rank 1 died from signal 15

Is this a priority/blocking bug?

• Priority

ETA

Intel can reproduce internally

[zhaomaosu]

The host ASAN runtime will intercept standard memory functions (mmap, munmap, etc.) with its own instrumented versions.
During MPI initialization, the CXI libfabric provider's memory hook mechanism (ofi_memhooks) also tries to intercept mmap by directly overwriting the function's machine code with a jump instruction (binary patching). This patching corrupts ASAN's version of mmap, causing subsequent calls to jump to an invalid address, resulting in a SEGV.
In short: both ASAN and libfabric's memhook try to hook mmap using incompatible mechanisms, and libfabric's patching destroys ASAN's hook.

There are two solutions for this issue:

1.Change default libfabric's memory monitoring from binary patching to the kernel's userfaultfd mechanism by setting env "FI_MR_CACHE_MONITOR=userfaultfd", which does not modify any code and is fully compatible with ASAN.

2.Hide ASAN's mmap symbol via linker version script to prevent libfabric from finding ASAN's mmap interceptor. It will instead patch libc's original mmap, leaving ASAN's instrumentation intact.

$ echo '{ local: mmap; mmap64; munmap; };' > hide_mmap.ver
$ mpicxx -Wl,--version-script=hide_mmap.ver  -fsycl -g -O1 -Xarch_host -fsanitize=address  Comp_GeoSeries_sycl.cpp -o Comp_GeoSeries_sycl-mpicxx_DP_ASAN_HOST

I recommend solution 1 since it's much easier to use.

[colleeneb]

Thank you for the clear explanation! We will add this to the user documentation. I don't think we'll change the default setting of FI_MR_CACHE_MONITOR right now, but we'll discuss internally.

[brianhol42]

@zhaomaosu This is probably (slightly) related to the Intel ticket. So I was able to use the environment variable and prevent the crash, which is great!

But at the end of the MPI program, I still get lots of MPI related output like:

Direct leak of 20 byte(s) in 1 object(s) allocated from:                                                                                                                                                       
    #0 0x0000004c67be in strdup /netbatch/donb38620_00/runDir/dir/workspace/NIT/xmain-rel/LX/xmainefi2linux_release/ws/icsws/llvm/compiler-rt/lib/asan/asan_interceptors.cpp:596:3
    #1 0x14b43bef9511 in pmix_hwloc_load_topology /tmp/rpmbuild/BUILD/pmix-4.2.9/src/hwloc/pmix_hwloc.c:671
    #2 0x14b43bf1a56b in _loadtp /tmp/rpmbuild/BUILD/pmix-4.2.9/src/client/pmix_client_topology.c:32
    #3 0x14b43bc8ecf0  (/usr/lib64/libevent_core-2.1.so.6+0x1dcf0) (BuildId: 8c9831e73fc1c55dd5d6e78c6f59c505306769a3)
    #4 0x14b43bc8f61e in event_base_loop (/usr/lib64/libevent_core-2.1.so.6+0x1e61e) (BuildId: 8c9831e73fc1c55dd5d6e78c6f59c505306769a3)
    #5 0x14b43bf5b8a0 in progress_engine /tmp/rpmbuild/BUILD/pmix-4.2.9/src/runtime/pmix_progress_threads.c:230
    #6 0x0000004dde3a in asan_thread_start(void*) /netbatch/donb38620_00/runDir/dir/workspace/NIT/xmain-rel/LX/xmainefi2linux_release/ws/icsws/llvm/compiler-rt/lib/asan/asan_interceptors.cpp:239:28
    #7 0x14b43c6d46e9 in start_thread (/lib64/libpthread.so.0+0xa6e9) (BuildId: 83611af746ef9652b7ecae972f50bec540256db0)
    #8 0x14b43c1d494e in clone (/lib64/libc.so.6+0x11794e) (BuildId: 171a59c1c43a8f7b93c3dff765aae0b675fe10f6)

But I added libmpi/cxi/fabric/pmix to my supp.txt file doesn't seem to suppress it. Is it possible to suppress these messages?

interceptor_via_lib:libsycl.so*
interceptor_via_lib:libur_loader.so*
interceptor_via_lib:libumf.so*
interceptor_via_lib:libomptarget.so*
interceptor_via_lib:libur_adapter*
interceptor_via_lib:libhwloc.so*
interceptor_via_lib:libze_*
interceptor_via_lib:libOpenCL*
interceptor_via_lib:libintelocl.so*
interceptor_via_lib:libintelocl.so*
interceptor_via_lib:libcommon_clang.so*
interceptor_via_lib:libtbb*
interceptor_via_lib:libcxi.so*
interceptor_via_lib:libpmix.so*
interceptor_via_lib:libmpi.so*
interceptor_via_lib:libfabric.so*

[zhaomaosu]

There is a memory leak in the program. If you think it's not a problem, you can disable leak sanitizer with runtime flag "ASAN_OPTIONS=detect_leaks=0"

ONEAPI_DEVICE_SELECTOR=level_zero:gpu ASAN_OPTIONS=detect_leaks=0 mpirun -n 2 --ppn 1 ./Comp_GeoSeries_sycl-mpicxx_DP_ASAN_HOST

[brianhol42]

@zhaomaosu Interesting. I think there must be several leaks with how the application memory is interacting with the MPI library. I'll see if i can find the source or else work with the MPICH team.

Much appreciated for all your insight.

[colleeneb]

I am closing this issue now that there is an update in the ALCF user guide. Let me know if we should reopen.