Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

CRASH and RANK ORDER VIOLATION with -exit_after_tracing #3346

Open
derekbruening opened this issue Jan 15, 2019 · 1 comment · May be fixed by #6362
Open

CRASH and RANK ORDER VIOLATION with -exit_after_tracing #3346

derekbruening opened this issue Jan 15, 2019 · 1 comment · May be fixed by #6362
Labels
Bug-DRCrash Component-DrMemtrace help wanted OpSys-Linux OpSys-x64

Comments

@derekbruening
Copy link
Contributor

When I was gathering the new test trace
clients/drcachesim/tests/drmemtrace.threadsig.x64.tracedir/drmemtrace.threadsig.10506.7343.trace.gz
I hit problems when I used -exit_after_tracing but only with certain values.
This only happens with a large -trace_after_instrs: so if we start tracing
near the end.

$ rm -rf drmemtrace.*.dir; gdb --args bin64/drrun -msgbox_mask 12 -t drcachesim -offline -trace_after_instrs 2180K -exit_after_tracing 120K -- ~/dr/test/threadsig 8 2000

add-symbol-file '/home/bruening/dr/git/build_x64_dbg_tests/bin64/../clients/lib64/debug/libdrmemtrace.so' 0x00007fa4afa97c10
add-symbol-file '/home/bruening/dr/git/build_x64_dbg_tests/ext/lib64/debug/libdrsyms.so' 0x00007fa4afcd0260
add-symbol-file '/home/bruening/dr/git/build_x64_dbg_tests/ext/lib64/debug/libdrwrap.so' 0x00007fa4aff18020
add-symbol-file '/home/bruening/dr/git/build_x64_dbg_tests/ext/lib64/debug/libdrmgr.so' 0x00007fa4b0126570
add-symbol-file '/home/bruening/dr/git/build_x64_dbg_tests/ext/lib64/debug/libdrutil.so' 0x00007fa4b03316a0
add-symbol-file '/home/bruening/dr/git/build_x64_dbg_tests/ext/lib64/debug/libdrcovlib.so' 0x00007fa4b0537dc0
add-symbol-file '/home/bruening/dr/git/build_x64_dbg_tests/ext/lib64/debug/libdrx.so' 0x00007fa4b07431f0
add-symbol-file '/home/bruening/dr/git/build_x64_dbg_tests/ext/lib64/debug/libdrreg.so' 0x00007fa4b094cb10
add-symbol-file '/lib/x86_64-linux-gnu/libgcc_s.so.1' 0x00007fa5329f4ac0
add-symbol-file '/lib64/ld-linux-x86-64.so.2' 0x00007fa532c0baa0
add-symbol-file '/lib/x86_64-linux-gnu/libc.so.6' 0x00007fa532e51910
add-symbol-file '/lib/x86_64-linux-gnu/libm.so.6' 0x00007fa5331d7680
add-symbol-file '/usr/lib/x86_64-linux-gnu/libstdc++.so.6' 0x00007fa533563090
add-symbol-file '/home/bruening/dr/git/build_x64_dbg_tests/lib64/debug/libdynamorio.so' 0x00007fa533ab3688

SIGSEGV
(gdb) bt
#0  __cxa_finalize (d=0x7fa4afcc0020) at cxa_finalize.c:56
#1  0x00007fa4afa97cc3 in __do_global_dtors_aux ()
#2  0x00007fa51fab29f0 in ?? ()
#3  0x00007fa533d49a48 in privload_call_lib_func (func=0x7fa51f96fe68) at /home/bruening/dr/git/src/core/unix/loader.c:883

<rank order violation shared_vm_areas(readwrite)@/home/bruening/dr/git/src/core/vmareas.c:1568 acquired after privload_lock(recursive)@/home/bruening/dr/git/src/core/loader_shared.c:61 in tid:390cf>

(gdb) bt
#0  0x00007fa533b5011f in deadlock_avoidance_lock (lock=0x7fa51f96c8a0, acquired=true, ownable=false) at /home/bruening/dr/git/src/core/utils.c:618
#1  0x00007fa533b51242 in read_lock (rw=0x7fa51f96c8a0) at /home/bruening/dr/git/src/core/utils.c:1225
#2  0x00007fa533c0bf1c in check_in_last_thread_vm_area (dcontext=0x7fa51f97ccc0, pc=0x0) at /home/bruening/dr/git/src/core/vmareas.c:8243
#3  0x00007fa533d3b90b in master_signal_handler_C (sig=11, siginfo=0x7fa51fb96af0, ucxt=0x7fa51fb969c0, xsp=0x7fa51fb969b8 "q\264\317\063\245\177")
    at /home/bruening/dr/git/src/core/unix/signal.c:5056
#4  0x00007fa533cfb471 in xfer_to_new_libdr () at /home/bruening/dr/git/src/core/arch/x86/x86.asm:1211

More info on the crash (from different run so addrs different):

(gdb) x/1i $pc
=> 0x7fcd95099c7d <__cxa_finalize+141>:	callq  *%rdx
(gdb) p /x $rdx
$1 = 0xa2ecc8dd87fdd68d

For the rank order: that lock acquisition looks bad in general. No, we
can't reorder to avoid this violation. We probably want to have
check_in_last_thread_vm_area() do a first read w/o the lock and only grab
it to confirm.
derekbruening added a commit that referenced this issue Jan 18, 2019
@derekbruening
i#3346 trace exit: avoid deadlock in check_in_last_thread_vm_area
3c4d0d3 
Removes the lock acquisition from check_in_last_thread_vm_area to
avoid deadlock.  The scenario we're trying to catch will read and
match with no extra lock due to the bb building lock.

It would be nice to have a test but it is not easy to write due to the
race that needs to be arranged.

Issue: #3346
@derekbruening derekbruening mentioned this issue Jan 18, 2019
Merged
derekbruening added a commit that referenced this issue Jan 18, 2019
@derekbruening
i#3346 trace exit: avoid deadlock in check_in_last_thread_vm_area (#3350
7003677 
)

Removes the lock acquisition from check_in_last_thread_vm_area to
avoid deadlock.  The scenario we're trying to catch will read and
match with no extra lock due to the bb building lock.

It would be nice to have a test but it is not easy to write due to the
race that needs to be arranged.

Issue: #3346
@prasun3
Copy link
Contributor

prasun3 commented May 23, 2023

I am seeing SIGSEGV with -exit_after_tracing. It is easily reproducible as long as it is not too small (before threads have spawned) and not too large (after "Estimation of pi" message is printed)


$ bin64/drrun -t drcachesim -offline -exit_after_tracing 100k -- suite/tests/bin/pthreads.pthreads
<Starting application /home/prasun/dynamorio-latest/build/suite/tests/bin/pthreads.pthreads (156497)>
<Initial options = -no_dynamic_options -client_lib '/home/prasun/dynamorio-latest/build/bin64/../clients/lib64/debug/libdrmemtrace.so;0;-offline -exit_after_tracing 100k' -client_lib64 '/home/prasun/dynamorio-latest/build/bin64/../clients/lib64/debug/libdrmemtrace.so;0;-offline -exit_after_tracing 100k' -code_api -stack_size 56K -signal_stack_size 32K -max_elide_jmp 0 -max_elide_call 0 -early_inject -emulate_brk -no_inline_ignored_syscalls -native_exec_default_list '' -no_native_exec_managed_code -no_indcall2direct >
<Paste into GDB to debug DynamoRIO clients:
set confirm off
add-symbol-file '/home/prasun/dynamorio-latest/build/bin64/../clients/lib64/debug/libdrmemtrace.so' 0x00007fffb3d64490
add-symbol-file '/home/prasun/dynamorio-latest/build/lib64/debug/libdynamorio.so' 0x000000007104b820
add-symbol-file '/home/prasun/dynamorio-latest/build/ext/lib64/debug/libdrsyms.so' 0x00007fffb3ff4260
add-symbol-file '/lib/x86_64-linux-gnu/libc.so.6' 0x00007ffff7950360
add-symbol-file '/lib64/ld-linux-x86-64.so.2' 0x00007ffff7702f10
add-symbol-file '/home/prasun/dynamorio-latest/build/ext/lib64/debug/libdrwrap.so' 0x00007fffb423c720
add-symbol-file '/home/prasun/dynamorio-latest/build/ext/lib64/debug/libdrmgr.so' 0x00007fffb444d090
add-symbol-file '/home/prasun/dynamorio-latest/build/ext/lib64/debug/libdrutil.so' 0x00007fffb465c850
add-symbol-file '/home/prasun/dynamorio-latest/build/ext/lib64/debug/libdrstatecmp.so' 0x00007fffb48627b0
add-symbol-file '/home/prasun/dynamorio-latest/build/ext/lib64/debug/libdrcovlib.so' 0x00007fffb4a68e60
add-symbol-file '/home/prasun/dynamorio-latest/build/ext/lib64/debug/libdrx.so' 0x00007fffb4c753a0
add-symbol-file '/home/prasun/dynamorio-latest/build/ext/lib64/debug/libdrreg.so' 0x00007fffb4e88010
add-symbol-file '/home/prasun/dynamorio-latest/build/ext/lib64/debug/libdrbbdup.so' 0x00007fffb5097340
add-symbol-file '/home/prasun/dynamorio-latest/build/ext/lib64/debug/libdrpttracer.so' 0x00007fffb52a41b0
add-symbol-file '/usr/lib/x86_64-linux-gnu/libsnappy.so.1' 0x00007ffff74fb1b0
add-symbol-file '/usr/lib/x86_64-linux-gnu/libstdc++.so.6' 0x00007ffff71879c0
add-symbol-file '/lib/x86_64-linux-gnu/libm.so.6' 0x00007ffff6d57a80
add-symbol-file '/lib/libgcc_s.so.1' 0x00007ffff6b35b50
add-symbol-file '/lib/x86_64-linux-gnu/libz.so.1' 0x00007ffff6916f90
>
<curiosity: rex.w on OPSZ_6_irex10_short4!>
<(1+x) Handling our fault in a TRY at 0x00000000712b29dd>
Exiting process after ~103291 references.
<Stopping application /home/prasun/dynamorio-latest/build/suite/tests/bin/pthreads.pthreads (156497)>
<Application /home/prasun/dynamorio-latest/build/suite/tests/bin/pthreads.pthreads (156497).  DynamoRIO Cache Simulator Tracer internal crash at PC 0x00007ffff7972603.  Please report this at http://dynamorio.org/issues.  Program aborted.
Received SIGSEGV at pc 0x00007ffff7972603 in thread 156498
Base: 0x0000000071000000
Registers:eax=0x00000000000000a0 ebx=0x00007ffdb3e4fc50 ecx=0x00007fffb3d64500 edx=0x380dc18a52d31d25
        esi=0x0000000000000000 edi=0x00007fffb3fab960 esp=0x00007ffdb3f825a0 ebp=0x00007ffff7d1f708
        r8 =0x000000000000294f r9 =0x00007fffb3fab960 r10=0x0000000000000003 r11=0x0000000000000246
        r12=0x00007ffdb3e4fbc0 r13=0x00007fffb3fa60a0 r14=0x00007ffdb3e4fbd0 r15=0x0000000000000065
        eflags=0x0000000000010246
version 9.93.19498, custom build
-no_dynamic_options -client_lib '/home/prasun/dynamorio-latest/build/bin64/../clients/lib64/debug/libdrmemtrace.so;0;-offline -exit_after_tracing 100k' -client_lib64 '/home/prasun/dynamorio-latest/build/bin64/../clients/lib64/debug/libdrmemtrace.so;0;-offline -exit_after_tracing 100k' -code_api -stack_size 56K -signal_stac
0x00007ffff7d1f708 0x0000000000000000>

@prasun3 prasun3 mentioned this issue Oct 12, 2023
Merged
prasun3 added a commit that referenced this issue Oct 12, 2023
@prasun3
i3346 test: Added a test that reproduces the crash
15afdf4 
Issue: #3346
@prasun3 prasun3 linked a pull request Oct 12, 2023 that will close this issue
i3346 test: Added a test that reproduces the crash #6362
Open
@prasun3 prasun3 linked a pull request Oct 12, 2023 that will close this issue
Open
prasun3 added a commit that referenced this issue Oct 13, 2023
@prasun3
i3346 test: Fix regex failure when crash doesn't occur
aaee847 
Issue: #3346
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
Bug-DRCrash Component-DrMemtrace help wanted OpSys-Linux OpSys-x64
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

i3346 test: Added a test that reproduces the crash DynamoRIO/dynamorio
2 participants
@prasun3 @derekbruening