Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Missing debugging symbols #130

Open
technicalpickles opened this issue Oct 13, 2023 · 1 comment
Open

Missing debugging symbols #130

technicalpickles opened this issue Oct 13, 2023 · 1 comment
Labels
bug Something isn't working

Comments

@technicalpickles
Copy link

Describe the bug

We're trying to diagnose some hung processes. We're able to use signal handling to get ruby backtraces (ie sidekiq's TTIN handler), but the problem seems to be one level deeper, ie C. The processes are multithreaded, so we need to get the state of every

SIGABRT/SIGEV are able to trigger ruby dumping it's own

These signals (and others) do also trigger a kernel core dump. If I try to analyze the core with gdb after the fact, it doesn't appear there are any debugging symbols present:

Reading symbols from /usr/lib/fullstaq-ruby/versions/3.2.2-jemalloc/bin/ruby...
(No debugging symbols found in /usr/lib/fullstaq-ruby/versions/3.2.2-jemalloc/bin/ruby)

To Reproduce
Linux distribution: Ubuntu 22.04
Package name: fullstaq-ruby-3.2.2

Steps to reproduce the behavior:

  1. Enable dumping cores: ulimit -c unlimited
  2. Run command ruby -e 'puts Process.pid; sleep 10000' &, then kill -s ABRT <pid>
  3. gdb $(which ruby) core (assuming kernel.core_pattern=core)

Note: this is running an x86_64 docker image on an M1, so I a fair amount of output is due to that rather than debugging symbols.

Reading symbols from /usr/lib/fullstaq-ruby/versions/3.2.2-jemalloc/bin/ruby...
(No debugging symbols found in /usr/lib/fullstaq-ruby/versions/3.2.2-jemalloc/bin/ruby)

warning: malformed note - filename area is too big

warning: Can't open file @ which was expanded to @ during file-backed mapping note processing

warning: Can't open file  which was expanded to  during file-backed mapping note processing

warning: Can't open file  which was expanded to  during file-backed mapping note processing

warning: Can't open file  which was expanded to  during file-backed mapping note processing

warning: Can't open file `@ which was expanded to `@ during file-backed mapping note processing

warning: Can't open file  which was expanded to  during file-backed mapping note processing

warning: Can't open file  which was expanded to  during file-backed mapping note processing

warning: Can't open file ( which was expanded to ( during file-backed mapping note processing

warning: Can't open file  which was expanded to  during file-backed mapping note processing

warning: Can't open file  which was expanded to  during file-backed mapping note processing

warning: Can't open file  which was expanded to  during file-backed mapping note processing

warning: Can't open file  which was expanded to  during file-backed mapping note processing

warning: Can't open file  which was expanded to  during file-backed mapping note processing

warning: Can't open file  which was expanded to  during file-backed mapping note processing

warning: Can't open file  which was expanded to  during file-backed mapping note processing

warning: Can't open file `@ which was expanded to `@ during file-backed mapping note processing

warning: Can't open file  which was expanded to  during file-backed mapping note processing

warning: Can't open file  which was expanded to  during file-backed mapping note processing

warning: Can't open file  which was expanded to  during file-backed mapping note processing

warning: Can't open file �!@ which was expanded to �!@ during file-backed mapping note processing

warning: Can't open file  which was expanded to  during file-backed mapping note processing

warning: Can't open file  which was expanded to  during file-backed mapping note processing

warning: Can't open file � which was expanded to � during file-backed mapping note processing

warning: Can't open file  which was expanded to  during file-backed mapping note processing

warning: Can't open file  which was expanded to  during file-backed mapping note processing

warning: Can't open file  which was expanded to  during file-backed mapping note processing

warning: Can't open file  which was expanded to  during file-backed mapping note processing

warning: Can't open file  which was expanded to  during file-backed mapping note processing

warning: Can't open file  which was expanded to  during file-backed mapping note processing

warning: Can't open file �!@ which was expanded to �!@ during file-backed mapping note processing

warning: Can't open file  which was expanded to  during file-backed mapping note processing

warning: Can't open file  which was expanded to  during file-backed mapping note processing

warning: Can't open file  which was expanded to  during file-backed mapping note processing

warning: Can't open file  "@ which was expanded to  "@ during file-backed mapping note processing

warning: Can't open file  which was expanded to  during file-backed mapping note processing

warning: Can't open file  which was expanded to  during file-backed mapping note processing

warning: Can't open file  which was expanded to  during file-backed mapping note processing

warning: Can't open file  which was expanded to  during file-backed mapping note processing

warning: Can't open file  which was expanded to  during file-backed mapping note processing

warning: Can't open file  which was expanded to  during file-backed mapping note processing

warning: Can't open file  which was expanded to  during file-backed mapping note processing

warning: Can't open file  which was expanded to  during file-backed mapping note processing

warning: Can't open file  which was expanded to  during file-backed mapping note processing

warning: Can't open file  "@ which was expanded to  "@ during file-backed mapping note processing

warning: Can't open file  which was expanded to  during file-backed mapping note processing

warning: Can't open file  which was expanded to  during file-backed mapping note processing

warning: Can't open file  which was expanded to  during file-backed mapping note processing

warning: Can't open file @"@ which was expanded to @"@ during file-backed mapping note processing

warning: Can't open file  which was expanded to  during file-backed mapping note processing

warning: Can't open file  which was expanded to  during file-backed mapping note processing

warning: Can't open file  which was expanded to  during file-backed mapping note processing

warning: Can't open file  which was expanded to  during file-backed mapping note processing

warning: Can't open file  which was expanded to  during file-backed mapping note processing

warning: Can't open file  which was expanded to  during file-backed mapping note processing

warning: Can't open file  which was expanded to  during file-backed mapping note processing

warning: Can't open file  which was expanded to  during file-backed mapping note processing

warning: Can't open file  which was expanded to  during file-backed mapping note processing

warning: Can't open file 0#@ which was expanded to 0#@ during file-backed mapping note processing

warning: Can't open file  which was expanded to  during file-backed mapping note processing

warning: Can't open file  which was expanded to  during file-backed mapping note processing

warning: Can't open file  which was expanded to  during file-backed mapping note processing

warning: Can't open file `#@ which was expanded to `#@ during file-backed mapping note processing

warning: Can't open file  which was expanded to  during file-backed mapping note processing

warning: Can't open file  which was expanded to  during file-backed mapping note processing

warning: Can't open file  which was expanded to  during file-backed mapping note processing

warning: Can't open file  which was expanded to  during file-backed mapping note processing

warning: Can't open file  which was expanded to  during file-backed mapping note processing

warning: Can't open file  which was expanded to  during file-backed mapping note processing

warning: Can't open file  which was expanded to  during file-backed mapping note processing

warning: Can't open file  which was expanded to  during file-backed mapping note processing

warning: Can't open file  which was expanded to  during file-backed mapping note processing

warning: Can't open file  which was expanded to  during file-backed mapping note processing

warning: Can't open file  which was expanded to  during file-backed mapping note processing

warning: Can't open file `#@ which was expanded to `#@ during file-backed mapping note processing

warning: Can't open file  which was expanded to  during file-backed mapping note processing

warning: Can't open file  which was expanded to  during file-backed mapping note processing

warning: Can't open file  which was expanded to  during file-backed mapping note processing

warning: Can't open file �$@ which was expanded to �$@ during file-backed mapping note processing

warning: Can't open file  which was expanded to  during file-backed mapping note processing

warning: Can't open file  which was expanded to  during file-backed mapping note processing

warning: Can't open file  which was expanded to  during file-backed mapping note processing

warning: Can't open file  which was expanded to  during file-backed mapping note processing

warning: Can't open file  which was expanded to  during file-backed mapping note processing

warning: Can't open file  which was expanded to  during file-backed mapping note processing

warning: Can't open file  which was expanded to  during file-backed mapping note processing

warning: Can't open file  which was expanded to  during file-backed mapping note processing

warning: Can't open file  which was expanded to  during file-backed mapping note processing

warning: Can't open file  which was expanded to  during file-backed mapping note processing

warning: Can't open file �$@ which was expanded to �$@ during file-backed mapping note processing

warning: core file may not match specified executable file.

warning: Selected architecture i386:x86-64 is not compatible with reported target architecture i386

warning: Architecture rejected target-supplied description

warning: Couldn't find general-purpose registers in core file.

warning: Unexpected size of section `.reg2' in core file.
Core was generated by `/usr/bin/qemu-x86_64 /usr/lib/fullstaq-ruby/versions/3.2.2-jemalloc/bin/ruby ru'.

warning: Couldn't find general-purpose registers in core file.

warning: Unexpected size of section `.reg2' in core file.
#0  <unavailable> in ?? ()
(gdb)

Expected behavior

After opening the core, being able to run bt and see the backtrace. This is what it looks like on the vanilla ruby:3.2.2 docker image:

GNU gdb (Debian 10.1-1.7) 10.1.90.20210103-git
Copyright (C) 2021 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
Type "show copying" and "show warranty" for details.
This GDB was configured as "aarch64-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<https://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
    <http://www.gnu.org/software/gdb/documentation/>.

For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from /usr/local/bin/ruby...
[New LWP 370]
[New LWP 371]
[New LWP 373]
[New LWP 372]
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/aarch64-linux-gnu/libthread_db.so.1".
Core was generated by `ruby reproducible-case.rb'.
Program terminated with signal SIGABRT, Aborted.
#0  0x0000fffface5b3c8 in __ppoll (fds=fds@entry=0xfffff4c4d190, nfds=nfds@entry=1, timeout=<optimized out>, timeout@entry=0x0, sigmask=sigmask@entry=0x0)
    at ../sysdeps/unix/sysv/linux/ppoll.c:44
44	../sysdeps/unix/sysv/linux/ppoll.c: No such file or directory.
[Current thread is 1 (Thread 0xffffacd6da00 (LWP 370))]
(gdb) bt
#0  0x0000fffface5b3c8 in __ppoll (fds=fds@entry=0xfffff4c4d190, nfds=nfds@entry=1, timeout=<optimized out>, timeout@entry=0x0, sigmask=sigmask@entry=0x0)
    at ../sysdeps/unix/sysv/linux/ppoll.c:44
#1  0x0000ffffad3ee258 in rb_sigwait_sleep (th=th@entry=0xaaaae7b964f0, sigwait_fd=sigwait_fd@entry=3, rel=rel@entry=0x0) at hrtime.h:158
#2  0x0000ffffad3ef6c8 [PAC] in native_sleep (th=th@entry=0xaaaae7b964f0, rel=0x0) at thread_pthread.c:2347
#3  0x0000ffffad3f3090 [PAC] in thread_join_sleep (arg=arg@entry=281474788283432) at thread.c:1056
#4  0x0000ffffad263d10 [PAC] in rb_ensure (b_proc=b_proc@entry=0xffffad3f2e90 <thread_join_sleep>, data1=data1@entry=281474788283432,
    e_proc=e_proc@entry=0xffffad3e7ee0 <remove_from_join_list>, data2=data2@entry=281474788283432) at eval.c:1007
#5  0x0000ffffad3eb0c8 [PAC] in thread_join (limit=0x0, timeout=4, target_th=0xaaaae7fa66f0) at thread.c:1105
#6  thread_join_m (argc=<optimized out>, argv=<optimized out>, self=<optimized out>) at thread.c:1202
#7  0x0000ffffad444770 [PAC] in vm_call0_cfunc_with_frame (argv=0xfffff4c4d650, calling=0xfffff4c4d4c8, ec=0xaaaae7b97680) at vm_eval.c:150
#8  vm_call0_cfunc (argv=0xfffff4c4d650, calling=0xfffff4c4d4c8, ec=0xaaaae7b97680) at vm_eval.c:164
#9  vm_call0_body (ec=ec@entry=0xaaaae7b97680, calling=calling@entry=0xfffff4c4d598, argv=argv@entry=0xfffff4c4d650) at vm_eval.c:210
#10 0x0000ffffad4462c8 [PAC] in vm_call0_cc (kw_splat=0, cc=0xffffa8466ad8, argv=0xfffff4c4d650, argc=0, id=13617, recv=281473504931160, ec=0xaaaae7b97680) at vm_eval.c:87
#11 rb_call0 (ec=0xaaaae7b97680, recv=281473504931160, mid=13617, argc=argc@entry=0, argv=argv@entry=0xfffff4c4d650, call_scope=call_scope@entry=CALL_PUBLIC, self=281473504931360)
    at vm_eval.c:551
#12 0x0000ffffad446fcc [PAC] in rb_call (scope=CALL_PUBLIC, argv=0xfffff4c4d650, argc=0, mid=<optimized out>, recv=<optimized out>) at vm_eval.c:877
#13 0x0000ffffad3d9524 in rb_sym_proc_call (mid=<optimized out>, argc=argc@entry=1, argv=argv@entry=0xfffff4c4d648, kw_splat=kw_splat@entry=0, passed_proc=passed_proc@entry=4)
    at string.c:11547
#14 0x0000ffffad4439dc in vm_yield_with_symbol (block_handler=0, kw_splat=0, argv=0xfffff4c4d648, argc=1, symbol=<optimized out>, ec=0xaaaae7b97680) at vm_insnhelper.c:4297
#15 invoke_block_from_c_bh (force_blockarg=<optimized out>, is_lambda=<optimized out>, cref=<optimized out>, passed_block_handler=<optimized out>, kw_splat=<optimized out>,
    argv=<optimized out>, argc=<optimized out>, block_handler=<optimized out>, ec=<optimized out>) at vm.c:1481
#16 vm_yield_with_cref (is_lambda=0, cref=0x0, kw_splat=0, argv=0xfffff4c4d648, argc=1, ec=0xaaaae7b97680) at vm.c:1509
#17 vm_yield (kw_splat=0, argv=0xfffff4c4d648, argc=1, ec=0xaaaae7b97680) at vm.c:1517
#18 rb_yield_0 (argv=0xfffff4c4d648, argc=1) at vm_eval.c:1348
#19 rb_yield (val=<optimized out>) at vm_eval.c:1364
#20 0x0000ffffad1c6bd4 [PAC] in rb_ary_each (ary=281473504931360) at array.c:2740
#21 0x0000ffffad42a120 [PAC] in vm_call_cfunc_with_frame (ec=0xaaaae7b97680, reg_cfp=0xffffacd16f90, calling=<optimized out>) at vm_insnhelper.c:3268
#22 0x0000ffffad4300f8 [PAC] in vm_sendish (ec=0xaaaae7b97680, reg_cfp=0xffffacd16f90, cd=0xaaaae7c522b0, block_handler=<optimized out>, method_explorer=mexp_search_method)
    at vm_callinfo.h:367
#23 0x0000ffffad439970 [PAC] in vm_exec_core (ec=0xaaaae7b97680, initial=1, initial@entry=0) at insns.def:801
#24 0x0000ffffad43f4fc [PAC] in rb_vm_exec (ec=0xaaaae7b97680, jit_enable_p=true) at vm.c:2383
#25 0x0000ffffad25f54c [PAC] in rb_ec_exec_node (ec=ec@entry=0xaaaae7b97680, n=n@entry=0xffffacc00b00) at eval.c:289
#26 0x0000ffffad2659bc [PAC] in ruby_run_node (n=0xffffacc00b00) at eval.c:330
#27 0x0000aaaab8f90b9c [PAC] in rb_main (argv=0xfffff4c4dd68, argc=2) at ./main.c:38
#28 main (argc=<optimized out>, argv=<optimized out>) at ./main.c:57

Additional context

Interestingly, I am able to get C level backtrace when using kill -s SEGV. However, it seems to only include the backtrace of the thread responding, not multiple threads.
@technicalpickles technicalpickles added the bug Something isn't working label Oct 13, 2023
@technicalpickles
Copy link
Author

I was looking at how ruby is built, and found this line:

server-edition/container-entrypoints/build-ruby

Lines 118 to 121 in 52afc2f

run strip --strip-all "$DESTDIR/$INSTALL_PREFIX/bin/ruby"
echo "+ find $DESTDIR/$INSTALL_PREFIX -name '*.so' -print0 | xargs -0 strip --strip-debug"
set -o pipefail
find "$DESTDIR/$INSTALL_PREFIX" -name '*.so' -print0 | xargs -0 strip --strip-debug

This explains where they went 😁

I'm not familiar enough with it, but some googling lead me to https://wiki.ubuntu.com/Debug%20Symbol%20Packages ... maybe creating dbg packages is an option?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug Something isn't working
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@technicalpickles