Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Segfault in TestDQMOnlineClient-visualization #42093

Open
makortel opened this issue Jun 26, 2023 · 20 comments
Open

Segfault in TestDQMOnlineClient-visualization #42093

makortel opened this issue Jun 26, 2023 · 20 comments

Comments

@makortel
Copy link
Contributor

The test TestDQMOnlineClient-visualization crashed in CMSSW_13_2_NONLTO_X_2023-06-24-1100 on el8_amd64_gcc11 with

Begin processing the 100th record. Run 355380, Event 26785259, LumiSection 19 on stream 4 at 24-Jun-2023 18:59:42.015 CEST

Thread 9 (Thread 0x14b8941fe700 (LWP 1590063) "cmsRun"):
#2  0x000014b8eb4b5590 in sig_pause_for_stacktrace () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02790/el8_amd64_gcc11/cms/cmssw-patch/CMSSW_13_2_NONLTO_X_2023-06-24-1100/lib/el8_amd64_gcc11/pluginFWCoreServicesPlugins.so
#3  <signal handler called>
#4  0x000014b8f34ff9bd in syscall () from /lib64/libc.so.6
#5  0x000014b8f472e2c4 in tbb::detail::r1::futex_wait (comparand=2, futex=0x14b8941f7cb0) at /data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/el8_amd64_gcc11/external/tbb/v2021.9.0-b1d97183c007769a478c02aa8504b8cb/tbb-v2021.9.0/src/tbb/semaphore.h:100
#6  tbb::detail::r1::binary_semaphore::P (this=0x14b8941f7cb0) at /data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/el8_amd64_gcc11/external/tbb/v2021.9.0-b1d97183c007769a478c02aa8504b8cb/tbb-v2021.9.0/src/tbb/semaphore.h:253
#7  tbb::detail::r1::sleep_node<tbb::detail::r1::address_context>::wait (this=0x14b8941f7c80) at /data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/el8_amd64_gcc11/external/tbb/v2021.9.0-b1d97183c007769a478c02aa8504b8cb/tbb-v2021.9.0/src/tbb/concurrent_monitor.h:170
#8  tbb::detail::r1::concurrent_monitor_base<tbb::detail::r1::address_context>::commit_wait (this=<optimized out>, node=...) at /data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/el8_amd64_gcc11/external/tbb/v2021.9.0-b1d97183c007769a478c02aa8504b8cb/tbb-v2021.9.0/src/tbb/concurrent_monitor.h:232
#9  tbb::detail::r1::concurrent_monitor_base<tbb::detail::r1::address_context>::commit_wait (node=..., this=0x14b8f4761460 <_ZN3tbb6detail2r1L20address_waiter_tableE.lto_priv.0+52320>) at /data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/el8_amd64_gcc11/external/tbb/v2021.9.0-b1d97183c007769a478c02aa8504b8cb/tbb-v2021.9.0/src/tbb/concurrent_monitor.h:228
#10 tbb::detail::r1::concurrent_monitor_base<tbb::detail::r1::address_context>::wait<tbb::detail::r1::sleep_node<tbb::detail::r1::address_context>, tbb::detail::d1::delegate_base&> (node=..., pred=..., this=0x14b8f4761460 <_ZN3tbb6detail2r1L20address_waiter_tableE.lto_priv.0+52320>) at /data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/el8_amd64_gcc11/external/tbb/v2021.9.0-b1d97183c007769a478c02aa8504b8cb/tbb-v2021.9.0/src/tbb/concurrent_monitor.h:262
#11 tbb::detail::r1::wait_on_address (address=<optimized out>, predicate=..., context=<optimized out>) at /data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/el8_amd64_gcc11/external/tbb/v2021.9.0-b1d97183c007769a478c02aa8504b8cb/tbb-v2021.9.0/src/tbb/address_waiter.cpp:71
#12 0x000014b8f473e95c in tbb::detail::d1::waitable_atomic<int>::wait_until (order=<optimized out>, context=787, expected=<optimized out>, this=0x14b8f1dbb098) at /data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/el8_amd64_gcc11/external/tbb/v2021.9.0-b1d97183c007769a478c02aa8504b8cb/tbb-v2021.9.0/src/tbb/../../include/oneapi/tbb/detail/_waitable_atomic.h:79
#13 tbb::detail::r1::market::adjust_demand (this=0x14b8f1dcb580, a=..., delta=-7, mandatory=<optimized out>) at /data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/el8_amd64_gcc11/external/tbb/v2021.9.0-b1d97183c007769a478c02aa8504b8cb/tbb-v2021.9.0/src/tbb/market.cpp:586
#14 0x000014b8f4740934 in tbb::detail::r1::arena::is_out_of_work() [clone .isra.0] (this=0x14b8f1dbaf00) at /data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/el8_amd64_gcc11/external/tbb/v2021.9.0-b1d97183c007769a478c02aa8504b8cb/tbb-v2021.9.0/src/tbb/arena.cpp:333
#15 0x000014b8f4730bad in tbb::detail::r1::waiter_base::pause (this=0x14b8941f7f30) at /data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/el8_amd64_gcc11/external/tbb/v2021.9.0-b1d97183c007769a478c02aa8504b8cb/tbb-v2021.9.0/src/tbb/waiters.h:36
#16 tbb::detail::r1::outermost_worker_waiter::pause (this=0x14b8941f7f30) at /data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/el8_amd64_gcc11/external/tbb/v2021.9.0-b1d97183c007769a478c02aa8504b8cb/tbb-v2021.9.0/src/tbb/waiters.h:69
#17 tbb::detail::r1::task_dispatcher::receive_or_steal_task<true, tbb::detail::r1::outermost_worker_waiter> (this=0x14b8f1dbbc00, tls=..., ed=..., waiter=..., isolation=0, fifo_allowed=<optimized out>, critical_allowed=<optimized out>) at /data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/el8_amd64_gcc11/external/tbb/v2021.9.0-b1d97183c007769a478c02aa8504b8cb/tbb-v2021.9.0/src/tbb/task_dispatcher.h:231
#18 0x000014b8f473cac0 in tbb::detail::r1::task_dispatcher::local_wait_for_all<false, tbb::detail::r1::outermost_worker_waiter> (t=0x0, waiter=..., this=0x14b8f1dbbc00) at /data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/el8_amd64_gcc11/external/tbb/v2021.9.0-b1d97183c007769a478c02aa8504b8cb/tbb-v2021.9.0/src/tbb/task_dispatcher.h:350
#19 tbb::detail::r1::task_dispatcher::local_wait_for_all<tbb::detail::r1::outermost_worker_waiter> (t=0x0, waiter=..., this=0x14b8f1dbbc00) at /data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/el8_amd64_gcc11/external/tbb/v2021.9.0-b1d97183c007769a478c02aa8504b8cb/tbb-v2021.9.0/src/tbb/task_dispatcher.h:458
#20 tbb::detail::r1::arena::process (tls=..., this=<optimized out>) at /data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/el8_amd64_gcc11/external/tbb/v2021.9.0-b1d97183c007769a478c02aa8504b8cb/tbb-v2021.9.0/src/tbb/arena.cpp:137
#21 tbb::detail::r1::market::process (this=<optimized out>, j=...) at /data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/el8_amd64_gcc11/external/tbb/v2021.9.0-b1d97183c007769a478c02aa8504b8cb/tbb-v2021.9.0/src/tbb/market.cpp:599
#22 0x000014b8f473ebb5 in tbb::detail::r1::rml::private_worker::run (this=0x14b8ec24e000) at /data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/el8_amd64_gcc11/external/tbb/v2021.9.0-b1d97183c007769a478c02aa8504b8cb/tbb-v2021.9.0/src/tbb/private_server.cpp:271
#23 tbb::detail::r1::rml::private_worker::thread_routine (arg=0x14b8ec24e000) at /data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/el8_amd64_gcc11/external/tbb/v2021.9.0-b1d97183c007769a478c02aa8504b8cb/tbb-v2021.9.0/src/tbb/private_server.cpp:221
#24 0x000014b8f38931ca in start_thread () from /lib64/libpthread.so.0
#25 0x000014b8f34ffe73 in clone () from /lib64/libc.so.6

Thread 8 (Thread 0x14b8965fe700 (LWP 1590062) "cmsRun"):
#2  0x000014b8eb4b5590 in sig_pause_for_stacktrace () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02790/el8_amd64_gcc11/cms/cmssw-patch/CMSSW_13_2_NONLTO_X_2023-06-24-1100/lib/el8_amd64_gcc11/pluginFWCoreServicesPlugins.so
#3  <signal handler called>
#4  0x000014b8f34ff12b in sched_yield () from /lib64/libc.so.6
#5  0x000014b8f4730b8a in __gthread_yield () at /data/cmsbld/jenkins/workspace/build-any-ib/w/el8_amd64_gcc11/external/gcc/11.4.1-30ebdc301ebd200f2ae0e3d880258e65/include/c++/11.4.1/x86_64-redhat-linux-gnu/bits/gthr-default.h:693
#6  std::this_thread::yield () at /data/cmsbld/jenkins/workspace/build-any-ib/w/el8_amd64_gcc11/external/gcc/11.4.1-30ebdc301ebd200f2ae0e3d880258e65/include/c++/11.4.1/bits/std_thread.h:329
#7  tbb::detail::r1::stealing_loop_backoff::pause (this=0x14b8965f7f38) at /data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/el8_amd64_gcc11/external/tbb/v2021.9.0-b1d97183c007769a478c02aa8504b8cb/tbb-v2021.9.0/src/tbb/scheduler_common.h:266
#8  tbb::detail::r1::waiter_base::pause (this=0x14b8965f7f30) at /data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/el8_amd64_gcc11/external/tbb/v2021.9.0-b1d97183c007769a478c02aa8504b8cb/tbb-v2021.9.0/src/tbb/waiters.h:35
#9  tbb::detail::r1::outermost_worker_waiter::pause (this=0x14b8965f7f30) at /data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/el8_amd64_gcc11/external/tbb/v2021.9.0-b1d97183c007769a478c02aa8504b8cb/tbb-v2021.9.0/src/tbb/waiters.h:69
#10 tbb::detail::r1::task_dispatcher::receive_or_steal_task<true, tbb::detail::r1::outermost_worker_waiter> (this=0x14b8f1dbbc80, tls=..., ed=..., waiter=..., isolation=0, fifo_allowed=<optimized out>, critical_allowed=<optimized out>) at /data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/el8_amd64_gcc11/external/tbb/v2021.9.0-b1d97183c007769a478c02aa8504b8cb/tbb-v2021.9.0/src/tbb/task_dispatcher.h:231
#11 0x000014b8f473cac0 in tbb::detail::r1::task_dispatcher::local_wait_for_all<false, tbb::detail::r1::outermost_worker_waiter> (t=0x0, waiter=..., this=0x14b8f1dbbc80) at /data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/el8_amd64_gcc11/external/tbb/v2021.9.0-b1d97183c007769a478c02aa8504b8cb/tbb-v2021.9.0/src/tbb/task_dispatcher.h:350
#12 tbb::detail::r1::task_dispatcher::local_wait_for_all<tbb::detail::r1::outermost_worker_waiter> (t=0x0, waiter=..., this=0x14b8f1dbbc80) at /data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/el8_amd64_gcc11/external/tbb/v2021.9.0-b1d97183c007769a478c02aa8504b8cb/tbb-v2021.9.0/src/tbb/task_dispatcher.h:458
#13 tbb::detail::r1::arena::process (tls=..., this=<optimized out>) at /data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/el8_amd64_gcc11/external/tbb/v2021.9.0-b1d97183c007769a478c02aa8504b8cb/tbb-v2021.9.0/src/tbb/arena.cpp:137
#14 tbb::detail::r1::market::process (this=<optimized out>, j=...) at /data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/el8_amd64_gcc11/external/tbb/v2021.9.0-b1d97183c007769a478c02aa8504b8cb/tbb-v2021.9.0/src/tbb/market.cpp:599
#15 0x000014b8f473ebb5 in tbb::detail::r1::rml::private_worker::run (this=0x14b8ec24df80) at /data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/el8_amd64_gcc11/external/tbb/v2021.9.0-b1d97183c007769a478c02aa8504b8cb/tbb-v2021.9.0/src/tbb/private_server.cpp:271
#16 tbb::detail::r1::rml::private_worker::thread_routine (arg=0x14b8ec24df80) at /data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/el8_amd64_gcc11/external/tbb/v2021.9.0-b1d97183c007769a478c02aa8504b8cb/tbb-v2021.9.0/src/tbb/private_server.cpp:221
#17 0x000014b8f38931ca in start_thread () from /lib64/libpthread.so.0
#18 0x000014b8f34ffe73 in clone () from /lib64/libc.so.6

Thread 7 (Thread 0x14b894bff700 (LWP 1590061) "cmsRun"):
#2  0x000014b8eb4b5590 in sig_pause_for_stacktrace () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02790/el8_amd64_gcc11/cms/cmssw-patch/CMSSW_13_2_NONLTO_X_2023-06-24-1100/lib/el8_amd64_gcc11/pluginFWCoreServicesPlugins.so
#3  <signal handler called>
#4  0x000014b8f34ff9bd in syscall () from /lib64/libc.so.6
#5  0x000014b8f473ee9e in tbb::detail::r1::futex_wait (comparand=2, futex=0x14b8ec24de24) at /data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/el8_amd64_gcc11/external/tbb/v2021.9.0-b1d97183c007769a478c02aa8504b8cb/tbb-v2021.9.0/src/tbb/semaphore.h:100
#6  tbb::detail::r1::binary_semaphore::P (this=0x14b8ec24de24) at /data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/el8_amd64_gcc11/external/tbb/v2021.9.0-b1d97183c007769a478c02aa8504b8cb/tbb-v2021.9.0/src/tbb/semaphore.h:253
#7  tbb::detail::r1::rml::internal::thread_monitor::wait (this=0x14b8ec24de20) at /data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/el8_amd64_gcc11/external/tbb/v2021.9.0-b1d97183c007769a478c02aa8504b8cb/tbb-v2021.9.0/src/tbb/rml_thread_monitor.h:235
#8  tbb::detail::r1::rml::private_worker::run (this=0x14b8ec24de00) at /data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/el8_amd64_gcc11/external/tbb/v2021.9.0-b1d97183c007769a478c02aa8504b8cb/tbb-v2021.9.0/src/tbb/private_server.cpp:273
#9  tbb::detail::r1::rml::private_worker::thread_routine (arg=0x14b8ec24de00) at /data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/el8_amd64_gcc11/external/tbb/v2021.9.0-b1d97183c007769a478c02aa8504b8cb/tbb-v2021.9.0/src/tbb/private_server.cpp:221
#10 0x000014b8f38931ca in start_thread () from /lib64/libpthread.so.0
#11 0x000014b8f34ffe73 in clone () from /lib64/libc.so.6

Thread 6 (Thread 0x14b895bfd700 (LWP 1590060) "cmsRun"):
#2  0x000014b8eb4b5590 in sig_pause_for_stacktrace () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02790/el8_amd64_gcc11/cms/cmssw-patch/CMSSW_13_2_NONLTO_X_2023-06-24-1100/lib/el8_amd64_gcc11/pluginFWCoreServicesPlugins.so
#3  <signal handler called>
#4  0x000014b8f34ff9bd in syscall () from /lib64/libc.so.6
#5  0x000014b8f473ee9e in tbb::detail::r1::futex_wait (comparand=2, futex=0x14b8ec24df24) at /data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/el8_amd64_gcc11/external/tbb/v2021.9.0-b1d97183c007769a478c02aa8504b8cb/tbb-v2021.9.0/src/tbb/semaphore.h:100
#6  tbb::detail::r1::binary_semaphore::P (this=0x14b8ec24df24) at /data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/el8_amd64_gcc11/external/tbb/v2021.9.0-b1d97183c007769a478c02aa8504b8cb/tbb-v2021.9.0/src/tbb/semaphore.h:253
#7  tbb::detail::r1::rml::internal::thread_monitor::wait (this=0x14b8ec24df20) at /data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/el8_amd64_gcc11/external/tbb/v2021.9.0-b1d97183c007769a478c02aa8504b8cb/tbb-v2021.9.0/src/tbb/rml_thread_monitor.h:235
#8  tbb::detail::r1::rml::private_worker::run (this=0x14b8ec24df00) at /data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/el8_amd64_gcc11/external/tbb/v2021.9.0-b1d97183c007769a478c02aa8504b8cb/tbb-v2021.9.0/src/tbb/private_server.cpp:273
#9  tbb::detail::r1::rml::private_worker::thread_routine (arg=0x14b8ec24df00) at /data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/el8_amd64_gcc11/external/tbb/v2021.9.0-b1d97183c007769a478c02aa8504b8cb/tbb-v2021.9.0/src/tbb/private_server.cpp:221
#10 0x000014b8f38931ca in start_thread () from /lib64/libpthread.so.0
#11 0x000014b8f34ffe73 in clone () from /lib64/libc.so.6

Thread 5 (Thread 0x14b8971ff700 (LWP 1590059) "cmsRun"):
#2  0x000014b8eb4b5590 in sig_pause_for_stacktrace () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02790/el8_amd64_gcc11/cms/cmssw-patch/CMSSW_13_2_NONLTO_X_2023-06-24-1100/lib/el8_amd64_gcc11/pluginFWCoreServicesPlugins.so
#3  <signal handler called>
#4  0x000014b8f34ff9bd in syscall () from /lib64/libc.so.6
#5  0x000014b8f473ee9e in tbb::detail::r1::futex_wait (comparand=2, futex=0x14b8ec24dea4) at /data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/el8_amd64_gcc11/external/tbb/v2021.9.0-b1d97183c007769a478c02aa8504b8cb/tbb-v2021.9.0/src/tbb/semaphore.h:100
#6  tbb::detail::r1::binary_semaphore::P (this=0x14b8ec24dea4) at /data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/el8_amd64_gcc11/external/tbb/v2021.9.0-b1d97183c007769a478c02aa8504b8cb/tbb-v2021.9.0/src/tbb/semaphore.h:253
#7  tbb::detail::r1::rml::internal::thread_monitor::wait (this=0x14b8ec24dea0) at /data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/el8_amd64_gcc11/external/tbb/v2021.9.0-b1d97183c007769a478c02aa8504b8cb/tbb-v2021.9.0/src/tbb/rml_thread_monitor.h:235
#8  tbb::detail::r1::rml::private_worker::run (this=0x14b8ec24de80) at /data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/el8_amd64_gcc11/external/tbb/v2021.9.0-b1d97183c007769a478c02aa8504b8cb/tbb-v2021.9.0/src/tbb/private_server.cpp:273
#9  tbb::detail::r1::rml::private_worker::thread_routine (arg=0x14b8ec24de80) at /data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/el8_amd64_gcc11/external/tbb/v2021.9.0-b1d97183c007769a478c02aa8504b8cb/tbb-v2021.9.0/src/tbb/private_server.cpp:221
#10 0x000014b8f38931ca in start_thread () from /lib64/libpthread.so.0
#11 0x000014b8f34ffe73 in clone () from /lib64/libc.so.6

Thread 4 (Thread 0x14b89839d700 (LWP 1590057) "cmsRun"):
#0  0x000014b8f35bf868 in nanosleep () from /lib64/libc.so.6
#1  0x000014b8f35bf76e in sleep () from /lib64/libc.so.6
#2  0x000014b8eb4b5590 in sig_pause_for_stacktrace () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02790/el8_amd64_gcc11/cms/cmssw-patch/CMSSW_13_2_NONLTO_X_2023-06-24-1100/lib/el8_amd64_gcc11/pluginFWCoreServicesPlugins.so
#3  <signal handler called>
#4  0x000014b8f34ff9bd in syscall () from /lib64/libc.so.6
#5  0x000014b8f473ee9e in tbb::detail::r1::futex_wait (comparand=2, futex=0x14b8ec24e124) at /data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/el8_amd64_gcc11/external/tbb/v2021.9.0-b1d97183c007769a478c02aa8504b8cb/tbb-v2021.9.0/src/tbb/semaphore.h:100
#6  tbb::detail::r1::binary_semaphore::P (this=0x14b8ec24e124) at /data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/el8_amd64_gcc11/external/tbb/v2021.9.0-b1d97183c007769a478c02aa8504b8cb/tbb-v2021.9.0/src/tbb/semaphore.h:253
#7  tbb::detail::r1::rml::internal::thread_monitor::wait (this=0x14b8ec24e120) at /data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/el8_amd64_gcc11/external/tbb/v2021.9.0-b1d97183c007769a478c02aa8504b8cb/tbb-v2021.9.0/src/tbb/rml_thread_monitor.h:235
#8  tbb::detail::r1::rml::private_worker::run (this=0x14b8ec24e100) at /data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/el8_amd64_gcc11/external/tbb/v2021.9.0-b1d97183c007769a478c02aa8504b8cb/tbb-v2021.9.0/src/tbb/private_server.cpp:273
#9  tbb::detail::r1::rml::private_worker::thread_routine (arg=0x14b8ec24e100) at /data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/el8_amd64_gcc11/external/tbb/v2021.9.0-b1d97183c007769a478c02aa8504b8cb/tbb-v2021.9.0/src/tbb/private_server.cpp:221
#10 0x000014b8f38931ca in start_thread () from /lib64/libpthread.so.0
#11 0x000014b8f34ffe73 in clone () from /lib64/libc.so.6

Thread 3 (Thread 0x14b898d9e700 (LWP 1590056) "cmsRun"):
#3  0x000014b8eb4b901b in sig_dostack_then_abort () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02790/el8_amd64_gcc11/cms/cmssw-patch/CMSSW_13_2_NONLTO_X_2023-06-24-1100/lib/el8_amd64_gcc11/pluginFWCoreServicesPlugins.so
#4  <signal handler called>
#5  tbb::detail::r1::spawn (t=..., ctx=...) at /data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/el8_amd64_gcc11/external/tbb/v2021.9.0-b1d97183c007769a478c02aa8504b8cb/tbb-v2021.9.0/src/tbb/task_dispatcher.cpp:36
#6  0x000014b8f5f9b01a in tbb::detail::d1::function_task<edm::WaitingTaskHolder::doneWaiting(std::__exception_ptr::exception_ptr)::{lambda()#1}>::execute(tbb::detail::d1::execution_data&) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02790/el8_amd64_gcc11/cms/cmssw-patch/CMSSW_13_2_NONLTO_X_2023-06-24-1100/lib/el8_amd64_gcc11/libFWCoreFramework.so
#7  0x000014b8f473c96f in tbb::detail::r1::task_dispatcher::local_wait_for_all<false, tbb::detail::r1::outermost_worker_waiter> (t=0x14b87f42bc00, waiter=..., this=0x14b8f1dbbb80) at /data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/el8_amd64_gcc11/external/tbb/v2021.9.0-b1d97183c007769a478c02aa8504b8cb/tbb-v2021.9.0/src/tbb/task_dispatcher.h:322
#8  tbb::detail::r1::task_dispatcher::local_wait_for_all<tbb::detail::r1::outermost_worker_waiter> (t=0x0, waiter=..., this=0x14b8f1dbbb80) at /data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/el8_amd64_gcc11/external/tbb/v2021.9.0-b1d97183c007769a478c02aa8504b8cb/tbb-v2021.9.0/src/tbb/task_dispatcher.h:458
#9  tbb::detail::r1::arena::process (tls=..., this=<optimized out>) at /data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/el8_amd64_gcc11/external/tbb/v2021.9.0-b1d97183c007769a478c02aa8504b8cb/tbb-v2021.9.0/src/tbb/arena.cpp:137
#10 tbb::detail::r1::market::process (this=<optimized out>, j=...) at /data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/el8_amd64_gcc11/external/tbb/v2021.9.0-b1d97183c007769a478c02aa8504b8cb/tbb-v2021.9.0/src/tbb/market.cpp:599
#11 0x000014b8f473ebb5 in tbb::detail::r1::rml::private_worker::run (this=0x14b8ec24e080) at /data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/el8_amd64_gcc11/external/tbb/v2021.9.0-b1d97183c007769a478c02aa8504b8cb/tbb-v2021.9.0/src/tbb/private_server.cpp:271
#12 tbb::detail::r1::rml::private_worker::thread_routine (arg=0x14b8ec24e080) at /data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/el8_amd64_gcc11/external/tbb/v2021.9.0-b1d97183c007769a478c02aa8504b8cb/tbb-v2021.9.0/src/tbb/private_server.cpp:221
#13 0x000014b8f38931ca in start_thread () from /lib64/libpthread.so.0
#14 0x000014b8f34ffe73 in clone () from /lib64/libc.so.6

Thread 1 (Thread 0x14b8f2a23640 (LWP 1589014) "cmsRun"):
#2  0x000014b8eb4b5590 in sig_pause_for_stacktrace () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02790/el8_amd64_gcc11/cms/cmssw-patch/CMSSW_13_2_NONLTO_X_2023-06-24-1100/lib/el8_amd64_gcc11/pluginFWCoreServicesPlugins.so
#3  <signal handler called>
#4  0x000014b8f34ff12b in sched_yield () from /lib64/libc.so.6
#5  0x000014b8f473e917 in __gthread_yield () at /data/cmsbld/jenkins/workspace/build-any-ib/w/el8_amd64_gcc11/external/gcc/11.4.1-30ebdc301ebd200f2ae0e3d880258e65/include/c++/11.4.1/x86_64-redhat-linux-gnu/bits/gthr-default.h:693
#6  std::this_thread::yield () at /data/cmsbld/jenkins/workspace/build-any-ib/w/el8_amd64_gcc11/external/gcc/11.4.1-30ebdc301ebd200f2ae0e3d880258e65/include/c++/11.4.1/bits/std_thread.h:329
#7  tbb::detail::d0::timed_spin_wait_until<tbb::detail::d1::waitable_atomic<int>::wait_until(int, unsigned long, std::memory_order)::{lambda()#1}>(tbb::detail::d1::waitable_atomic<int>::wait_until(int, unsigned long, std::memory_order)::{lambda()#1}) (condition=...) at /data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/el8_amd64_gcc11/external/tbb/v2021.9.0-b1d97183c007769a478c02aa8504b8cb/tbb-v2021.9.0/src/tbb/../../include/oneapi/tbb/detail/_utils.h:129
#8  tbb::detail::d1::waitable_atomic<int>::wait_until (order=<optimized out>, context=788, expected=<optimized out>, this=0x14b8f1dbb098) at /data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/el8_amd64_gcc11/external/tbb/v2021.9.0-b1d97183c007769a478c02aa8504b8cb/tbb-v2021.9.0/src/tbb/../../include/oneapi/tbb/detail/_waitable_atomic.h:74
#9  tbb::detail::r1::market::adjust_demand (this=0x14b8f1dcb580, a=..., delta=7, mandatory=<optimized out>) at /data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/el8_amd64_gcc11/external/tbb/v2021.9.0-b1d97183c007769a478c02aa8504b8cb/tbb-v2021.9.0/src/tbb/market.cpp:586
#10 0x000014b8f474273b in tbb::detail::r1::arena::advertise_new_work<(tbb::detail::r1::arena::new_work_type)0> (this=0x14b8f1dbaf00) at /data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/el8_amd64_gcc11/external/tbb/v2021.9.0-b1d97183c007769a478c02aa8504b8cb/tbb-v2021.9.0/src/tbb/arena.h:547
#11 0x000014b8f5fcd8c8 in edm::EventProcessor::endUnfinishedLumi(bool) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02790/el8_amd64_gcc11/cms/cmssw-patch/CMSSW_13_2_NONLTO_X_2023-06-24-1100/lib/el8_amd64_gcc11/libFWCoreFramework.so
#12 0x000014b8f5fcd987 in edm::(anonymous namespace)::RunResources::~RunResources() () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02790/el8_amd64_gcc11/cms/cmssw-patch/CMSSW_13_2_NONLTO_X_2023-06-24-1100/lib/el8_amd64_gcc11/libFWCoreFramework.so
#13 0x000014b8f5fd26bd in edm::EventProcessor::runToCompletion() () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02790/el8_amd64_gcc11/cms/cmssw-patch/CMSSW_13_2_NONLTO_X_2023-06-24-1100/lib/el8_amd64_gcc11/libFWCoreFramework.so
#14 0x000000000040a1bd in tbb::detail::d1::task_arena_function<main::{lambda()#1}::operator()() const::{lambda()#1}, void>::operator()() const ()
#15 0x000014b8f4731877 in tbb::detail::r1::task_arena_impl::execute (ta=..., d=...) at /data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/el8_amd64_gcc11/external/tbb/v2021.9.0-b1d97183c007769a478c02aa8504b8cb/tbb-v2021.9.0/src/tbb/arena.cpp:688
#16 0x000000000040b009 in main::{lambda()#1}::operator()() const ()
#17 0x000000000040971c in main ()

Current Modules:

Module: none (crashed)/data/cmsbld/jenkins/workspace/ib-run-qa/CMSSW_13_2_NONLTO_X_2023-06-24-1100/src/DQM/Integration/test/runtest.sh: line 19: 1589014 Segmentation fault      (core dumped) cmsRun $CLIENTS_DIR/$1 unitTest=True

---> test TestDQMOnlineClient-visualization had ERRORS

https://cmssdt.cern.ch/SDT/cgi-bin/logreader/el8_amd64_gcc11/CMSSW_13_2_NONLTO_X_2023-06-24-1100/unitTestLogs/DQM/Integration#/

@makortel
Copy link
Contributor Author

assign core

FYI @cms-sw/dqm-l2

@cmsbuild
Copy link
Contributor

cmsbuild commented Jun 26, 2023

A new Issue was created by @makortel Matti Kortelainen.

@Dr15Jones, @perrotta, @dpiparo, @rappoccio, @makortel, @smuzaffar can you please review it and eventually sign/assign? Thanks.

cms-bot commands are listed here

@cmsbuild
Copy link
Contributor

New categories assigned: core

@Dr15Jones,@smuzaffar,@makortel you have been requested to review this Pull request/Issue and eventually sign? Thanks

@makortel
Copy link
Contributor Author

We updated TBB to 2021.9.0 in CMSSW_13_2_X_2023-06-20-2300 (with cms-sw/cmsdist#8553), it may be related.

@makortel
Copy link
Contributor Author

makortel commented Sep 6, 2023

Another one in CMSSW_13_3_X_2023-09-05-1100 on slc7_amd64_gcc11

Begin processing the 100th record. Run 355380, Event 26785259, LumiSection 19 on stream 4 at 05-Sep-2023 14:24:24.917 CEST

Thread 10 (Thread 0x14d71e3fc700 (LWP 1862305) "cmsRun"):
#2  0x000014d7802d0b10 in sig_pause_for_stacktrace () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02801/slc7_amd64_gcc11/cms/cmssw/CMSSW_13_3_X_2023-09-03-2300/lib/slc7_amd64_gcc11/pluginFWCoreServicesPlugins.so
#3  <signal handler called>
#4  0x000014d788646e29 in syscall () from /lib64/libc.so.6
#5  0x000014d789759eae in tbb::detail::r1::futex_wait (comparand=2, futex=0x14d78115df24) at /data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/slc7_amd64_gcc11/external/tbb/v2021.9.0-5caec19d753f5e33a53af823382dc5ea/tbb-v2021.9.0/src/tbb/semaphore.h:100
#6  tbb::detail::r1::binary_semaphore::P (this=0x14d78115df24) at /data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/slc7_amd64_gcc11/external/tbb/v2021.9.0-5caec19d753f5e33a53af823382dc5ea/tbb-v2021.9.0/src/tbb/semaphore.h:253
#7  tbb::detail::r1::rml::internal::thread_monitor::wait (this=0x14d78115df20) at /data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/slc7_amd64_gcc11/external/tbb/v2021.9.0-5caec19d753f5e33a53af823382dc5ea/tbb-v2021.9.0/src/tbb/rml_thread_monitor.h:235
#8  tbb::detail::r1::rml::private_worker::run (this=0x14d78115df00) at /data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/slc7_amd64_gcc11/external/tbb/v2021.9.0-5caec19d753f5e33a53af823382dc5ea/tbb-v2021.9.0/src/tbb/private_server.cpp:273
#9  tbb::detail::r1::rml::private_worker::thread_routine (arg=0x14d78115df00) at /data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/slc7_amd64_gcc11/external/tbb/v2021.9.0-5caec19d753f5e33a53af823382dc5ea/tbb-v2021.9.0/src/tbb/private_server.cpp:221
#10 0x000014d788923ea5 in start_thread () from /lib64/libpthread.so.0
#11 0x000014d78864cb0d in clone () from /lib64/libc.so.6

Thread 9 (Thread 0x14d71edfd700 (LWP 1862304) "cmsRun"):
#2  0x000014d7802d0b10 in sig_pause_for_stacktrace () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02801/slc7_amd64_gcc11/cms/cmssw/CMSSW_13_3_X_2023-09-03-2300/lib/slc7_amd64_gcc11/pluginFWCoreServicesPlugins.so
#3  <signal handler called>
#4  0x000014d788646e29 in syscall () from /lib64/libc.so.6
#5  0x000014d789759eae in tbb::detail::r1::futex_wait (comparand=2, futex=0x14d78115de24) at /data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/slc7_amd64_gcc11/external/tbb/v2021.9.0-5caec19d753f5e33a53af823382dc5ea/tbb-v2021.9.0/src/tbb/semaphore.h:100
#6  tbb::detail::r1::binary_semaphore::P (this=0x14d78115de24) at /data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/slc7_amd64_gcc11/external/tbb/v2021.9.0-5caec19d753f5e33a53af823382dc5ea/tbb-v2021.9.0/src/tbb/semaphore.h:253
#7  tbb::detail::r1::rml::internal::thread_monitor::wait (this=0x14d78115de20) at /data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/slc7_amd64_gcc11/external/tbb/v2021.9.0-5caec19d753f5e33a53af823382dc5ea/tbb-v2021.9.0/src/tbb/rml_thread_monitor.h:235
#8  tbb::detail::r1::rml::private_worker::run (this=0x14d78115de00) at /data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/slc7_amd64_gcc11/external/tbb/v2021.9.0-5caec19d753f5e33a53af823382dc5ea/tbb-v2021.9.0/src/tbb/private_server.cpp:273
#9  tbb::detail::r1::rml::private_worker::thread_routine (arg=0x14d78115de00) at /data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/slc7_amd64_gcc11/external/tbb/v2021.9.0-5caec19d753f5e33a53af823382dc5ea/tbb-v2021.9.0/src/tbb/private_server.cpp:221
#10 0x000014d788923ea5 in start_thread () from /lib64/libpthread.so.0
#11 0x000014d78864cb0d in clone () from /lib64/libc.so.6

Thread 8 (Thread 0x14d71f9fe700 (LWP 1862303) "cmsRun"):
#2  0x000014d7802d0b10 in sig_pause_for_stacktrace () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02801/slc7_amd64_gcc11/cms/cmssw/CMSSW_13_3_X_2023-09-03-2300/lib/slc7_amd64_gcc11/pluginFWCoreServicesPlugins.so
#3  <signal handler called>
#4  0x000014d788646e29 in syscall () from /lib64/libc.so.6
#5  0x000014d789759eae in tbb::detail::r1::futex_wait (comparand=2, futex=0x14d78115dea4) at /data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/slc7_amd64_gcc11/external/tbb/v2021.9.0-5caec19d753f5e33a53af823382dc5ea/tbb-v2021.9.0/src/tbb/semaphore.h:100
#6  tbb::detail::r1::binary_semaphore::P (this=0x14d78115dea4) at /data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/slc7_amd64_gcc11/external/tbb/v2021.9.0-5caec19d753f5e33a53af823382dc5ea/tbb-v2021.9.0/src/tbb/semaphore.h:253
#7  tbb::detail::r1::rml::internal::thread_monitor::wait (this=0x14d78115dea0) at /data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/slc7_amd64_gcc11/external/tbb/v2021.9.0-5caec19d753f5e33a53af823382dc5ea/tbb-v2021.9.0/src/tbb/rml_thread_monitor.h:235
#8  tbb::detail::r1::rml::private_worker::run (this=0x14d78115de80) at /data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/slc7_amd64_gcc11/external/tbb/v2021.9.0-5caec19d753f5e33a53af823382dc5ea/tbb-v2021.9.0/src/tbb/private_server.cpp:273
#9  tbb::detail::r1::rml::private_worker::thread_routine (arg=0x14d78115de80) at /data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/slc7_amd64_gcc11/external/tbb/v2021.9.0-5caec19d753f5e33a53af823382dc5ea/tbb-v2021.9.0/src/tbb/private_server.cpp:221
#10 0x000014d788923ea5 in start_thread () from /lib64/libpthread.so.0
#11 0x000014d78864cb0d in clone () from /lib64/libc.so.6

Thread 7 (Thread 0x14d7203ff700 (LWP 1862302) "cmsRun"):
#2  0x000014d7802d0b10 in sig_pause_for_stacktrace () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02801/slc7_amd64_gcc11/cms/cmssw/CMSSW_13_3_X_2023-09-03-2300/lib/slc7_amd64_gcc11/pluginFWCoreServicesPlugins.so
#3  <signal handler called>
#4  0x000014d788646e29 in syscall () from /lib64/libc.so.6
#5  0x000014d789759eae in tbb::detail::r1::futex_wait (comparand=2, futex=0x14d78115e024) at /data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/slc7_amd64_gcc11/external/tbb/v2021.9.0-5caec19d753f5e33a53af823382dc5ea/tbb-v2021.9.0/src/tbb/semaphore.h:100
#6  tbb::detail::r1::binary_semaphore::P (this=0x14d78115e024) at /data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/slc7_amd64_gcc11/external/tbb/v2021.9.0-5caec19d753f5e33a53af823382dc5ea/tbb-v2021.9.0/src/tbb/semaphore.h:253
#7  tbb::detail::r1::rml::internal::thread_monitor::wait (this=0x14d78115e020) at /data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/slc7_amd64_gcc11/external/tbb/v2021.9.0-5caec19d753f5e33a53af823382dc5ea/tbb-v2021.9.0/src/tbb/rml_thread_monitor.h:235
#8  tbb::detail::r1::rml::private_worker::run (this=0x14d78115e000) at /data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/slc7_amd64_gcc11/external/tbb/v2021.9.0-5caec19d753f5e33a53af823382dc5ea/tbb-v2021.9.0/src/tbb/private_server.cpp:273
#9  tbb::detail::r1::rml::private_worker::thread_routine (arg=0x14d78115e000) at /data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/slc7_amd64_gcc11/external/tbb/v2021.9.0-5caec19d753f5e33a53af823382dc5ea/tbb-v2021.9.0/src/tbb/private_server.cpp:221
#10 0x000014d788923ea5 in start_thread () from /lib64/libpthread.so.0
#11 0x000014d78864cb0d in clone () from /lib64/libc.so.6

Thread 6 (Thread 0x14d7215ff700 (LWP 1862301) "cmsRun"):
#2  0x000014d7802d0b10 in sig_pause_for_stacktrace () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02801/slc7_amd64_gcc11/cms/cmssw/CMSSW_13_3_X_2023-09-03-2300/lib/slc7_amd64_gcc11/pluginFWCoreServicesPlugins.so
#3  <signal handler called>
#4  0x000014d788646e29 in syscall () from /lib64/libc.so.6
#5  0x000014d789759eae in tbb::detail::r1::futex_wait (comparand=2, futex=0x14d78115dfa4) at /data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/slc7_amd64_gcc11/external/tbb/v2021.9.0-5caec19d753f5e33a53af823382dc5ea/tbb-v2021.9.0/src/tbb/semaphore.h:100
#6  tbb::detail::r1::binary_semaphore::P (this=0x14d78115dfa4) at /data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/slc7_amd64_gcc11/external/tbb/v2021.9.0-5caec19d753f5e33a53af823382dc5ea/tbb-v2021.9.0/src/tbb/semaphore.h:253
#7  tbb::detail::r1::rml::internal::thread_monitor::wait (this=0x14d78115dfa0) at /data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/slc7_amd64_gcc11/external/tbb/v2021.9.0-5caec19d753f5e33a53af823382dc5ea/tbb-v2021.9.0/src/tbb/rml_thread_monitor.h:235
#8  tbb::detail::r1::rml::private_worker::run (this=0x14d78115df80) at /data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/slc7_amd64_gcc11/external/tbb/v2021.9.0-5caec19d753f5e33a53af823382dc5ea/tbb-v2021.9.0/src/tbb/private_server.cpp:273
#9  tbb::detail::r1::rml::private_worker::thread_routine (arg=0x14d78115df80) at /data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/slc7_amd64_gcc11/external/tbb/v2021.9.0-5caec19d753f5e33a53af823382dc5ea/tbb-v2021.9.0/src/tbb/private_server.cpp:221
#10 0x000014d788923ea5 in start_thread () from /lib64/libpthread.so.0
#11 0x000014d78864cb0d in clone () from /lib64/libc.so.6

Thread 5 (Thread 0x14d7225f6700 (LWP 1862300) "cmsRun"):
#2  0x000014d7802d0b10 in sig_pause_for_stacktrace () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02801/slc7_amd64_gcc11/cms/cmssw/CMSSW_13_3_X_2023-09-03-2300/lib/slc7_amd64_gcc11/pluginFWCoreServicesPlugins.so
#3  <signal handler called>
#4  0x000014d788646e29 in syscall () from /lib64/libc.so.6
#5  0x000014d789759eae in tbb::detail::r1::futex_wait (comparand=2, futex=0x14d78115e124) at /data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/slc7_amd64_gcc11/external/tbb/v2021.9.0-5caec19d753f5e33a53af823382dc5ea/tbb-v2021.9.0/src/tbb/semaphore.h:100
#6  tbb::detail::r1::binary_semaphore::P (this=0x14d78115e124) at /data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/slc7_amd64_gcc11/external/tbb/v2021.9.0-5caec19d753f5e33a53af823382dc5ea/tbb-v2021.9.0/src/tbb/semaphore.h:253
#7  tbb::detail::r1::rml::internal::thread_monitor::wait (this=0x14d78115e120) at /data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/slc7_amd64_gcc11/external/tbb/v2021.9.0-5caec19d753f5e33a53af823382dc5ea/tbb-v2021.9.0/src/tbb/rml_thread_monitor.h:235
#8  tbb::detail::r1::rml::private_worker::run (this=0x14d78115e100) at /data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/slc7_amd64_gcc11/external/tbb/v2021.9.0-5caec19d753f5e33a53af823382dc5ea/tbb-v2021.9.0/src/tbb/private_server.cpp:273
#9  tbb::detail::r1::rml::private_worker::thread_routine (arg=0x14d78115e100) at /data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/slc7_amd64_gcc11/external/tbb/v2021.9.0-5caec19d753f5e33a53af823382dc5ea/tbb-v2021.9.0/src/tbb/private_server.cpp:221
#10 0x000014d788923ea5 in start_thread () from /lib64/libpthread.so.0
#11 0x000014d78864cb0d in clone () from /lib64/libc.so.6

Thread 4 (Thread 0x14d722ff7700 (LWP 1862299) "cmsRun"):
#2  0x000014d7802d0b10 in sig_pause_for_stacktrace () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02801/slc7_amd64_gcc11/cms/cmssw/CMSSW_13_3_X_2023-09-03-2300/lib/slc7_amd64_gcc11/pluginFWCoreServicesPlugins.so
#3  <signal handler called>
#4  0x000014d788646e29 in syscall () from /lib64/libc.so.6
#5  0x000014d789759eae in tbb::detail::r1::futex_wait (comparand=2, futex=0x14d78115e0a4) at /data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/slc7_amd64_gcc11/external/tbb/v2021.9.0-5caec19d753f5e33a53af823382dc5ea/tbb-v2021.9.0/src/tbb/semaphore.h:100
#6  tbb::detail::r1::binary_semaphore::P (this=0x14d78115e0a4) at /data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/slc7_amd64_gcc11/external/tbb/v2021.9.0-5caec19d753f5e33a53af823382dc5ea/tbb-v2021.9.0/src/tbb/semaphore.h:253
#7  tbb::detail::r1::rml::internal::thread_monitor::wait (this=0x14d78115e0a0) at /data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/slc7_amd64_gcc11/external/tbb/v2021.9.0-5caec19d753f5e33a53af823382dc5ea/tbb-v2021.9.0/src/tbb/rml_thread_monitor.h:235
#8  tbb::detail::r1::rml::private_worker::run (this=0x14d78115e080) at /data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/slc7_amd64_gcc11/external/tbb/v2021.9.0-5caec19d753f5e33a53af823382dc5ea/tbb-v2021.9.0/src/tbb/private_server.cpp:273
#9  tbb::detail::r1::rml::private_worker::thread_routine (arg=0x14d78115e080) at /data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/slc7_amd64_gcc11/external/tbb/v2021.9.0-5caec19d753f5e33a53af823382dc5ea/tbb-v2021.9.0/src/tbb/private_server.cpp:221
#10 0x000014d788923ea5 in start_thread () from /lib64/libpthread.so.0
#11 0x000014d78864cb0d in clone () from /lib64/libc.so.6

Thread 3 (Thread 0x14d76065a700 (LWP 1862126) "cmsRun"):
#0  0x000014d78892b1d9 in waitpid () from /lib64/libpthread.so.0
#1  0x000014d7802d0e87 in edm::service::cmssw_stacktrace_fork() () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02801/slc7_amd64_gcc11/cms/cmssw/CMSSW_13_3_X_2023-09-03-2300/lib/slc7_amd64_gcc11/pluginFWCoreServicesPlugins.so
#2  0x000014d7802d381a in edm::service::InitRootHandlers::stacktraceHelperThread() () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02801/slc7_amd64_gcc11/cms/cmssw/CMSSW_13_3_X_2023-09-03-2300/lib/slc7_amd64_gcc11/pluginFWCoreServicesPlugins.so
#3  0x000014d788f28b04 in std::execute_native_thread_routine (__p=0x14d76d992530) at ../../../../../libstdc++-v3/src/c++11/thread.cc:82
#4  0x000014d788923ea5 in start_thread () from /lib64/libpthread.so.0
#5  0x000014d78864cb0d in clone () from /lib64/libc.so.6

Thread 1 (Thread 0x14d78678c780 (LWP 1861056) "cmsRun"):
#3  0x000014d7802d434b in sig_dostack_then_abort () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02801/slc7_amd64_gcc11/cms/cmssw/CMSSW_13_3_X_2023-09-03-2300/lib/slc7_amd64_gcc11/pluginFWCoreServicesPlugins.so
#4  <signal handler called>
#5  tbb::detail::r1::spawn (t=..., ctx=...) at /data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/slc7_amd64_gcc11/external/tbb/v2021.9.0-5caec19d753f5e33a53af823382dc5ea/tbb-v2021.9.0/src/tbb/task_dispatcher.cpp:36
#6  0x000014d78aed69d9 in tbb::detail::d1::function_task<edm::WaitingTaskHolder::doneWaiting(std::__exception_ptr::exception_ptr)::{lambda()#1}>::execute(tbb::detail::d1::execution_data&) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02801/slc7_amd64_gcc11/cms/cmssw/CMSSW_13_3_X_2023-09-03-2300/lib/slc7_amd64_gcc11/libFWCoreFramework.so
#7  0x000014d78975fa7d in tbb::detail::r1::task_dispatcher::local_wait_for_all<false, tbb::detail::r1::external_waiter> (waiter=..., t=0x14d6e1939d00, this=0x14d785bbb900) at /data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/slc7_amd64_gcc11/external/tbb/v2021.9.0-5caec19d753f5e33a53af823382dc5ea/tbb-v2021.9.0/src/tbb/task_dispatcher.h:322
#8  tbb::detail::r1::task_dispatcher::local_wait_for_all<tbb::detail::r1::external_waiter> (waiter=..., t=<optimized out>, this=0x14d785bbb900) at /data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/slc7_amd64_gcc11/external/tbb/v2021.9.0-5caec19d753f5e33a53af823382dc5ea/tbb-v2021.9.0/src/tbb/task_dispatcher.h:458
#9  tbb::detail::r1::task_dispatcher::execute_and_wait (t=<optimized out>, wait_ctx=..., w_ctx=...) at /data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/slc7_amd64_gcc11/external/tbb/v2021.9.0-5caec19d753f5e33a53af823382dc5ea/tbb-v2021.9.0/src/tbb/task_dispatcher.cpp:168
#10 0x000014d78aeea42d in edm::FinalWaitingTask::wait() () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02801/slc7_amd64_gcc11/cms/cmssw/CMSSW_13_3_X_2023-09-03-2300/lib/slc7_amd64_gcc11/libFWCoreFramework.so
#11 0x000014d78aef6153 in edm::EventProcessor::endUnfinishedRun(bool) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02801/slc7_amd64_gcc11/cms/cmssw/CMSSW_13_3_X_2023-09-03-2300/lib/slc7_amd64_gcc11/libFWCoreFramework.so
#12 0x000014d78aef6213 in edm::(anonymous namespace)::RunResources::~RunResources() () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02801/slc7_amd64_gcc11/cms/cmssw/CMSSW_13_3_X_2023-09-03-2300/lib/slc7_amd64_gcc11/libFWCoreFramework.so
#13 0x000014d78aef83fd in edm::EventProcessor::runToCompletion() () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02801/slc7_amd64_gcc11/cms/cmssw/CMSSW_13_3_X_2023-09-03-2300/lib/slc7_amd64_gcc11/libFWCoreFramework.so
#14 0x000000000040c17d in tbb::detail::d1::task_arena_function<main::{lambda()#1}::operator()() const::{lambda()#1}, void>::operator()() const ()
#15 0x000014d78974c887 in tbb::detail::r1::task_arena_impl::execute (ta=..., d=...) at /data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/slc7_amd64_gcc11/external/tbb/v2021.9.0-5caec19d753f5e33a53af823382dc5ea/tbb-v2021.9.0/src/tbb/arena.cpp:688
#16 0x00000000004101e0 in main::{lambda()#1}::operator()() const ()
#17 0x0000000000407d7c in main ()

Current Modules:
Module: none

---> test TestDQMOnlineClient-visualization_secondInstance had ERRORS

https://cmssdt.cern.ch/SDT/cgi-bin/logreader/slc7_amd64_gcc11/CMSSW_13_3_X_2023-09-05-1100/unitTestLogs/DQM/Integration#/

@aandvalenzuela
Copy link
Contributor

Another one in CMSSW_14_1_X_2024-02-21-2300 on el8_amd64_gcc12:

Thread 3 (Thread 0x148615bcb700 (LWP 2694561) "cmsRun"):
#0  0x00001486631d3918 in nanosleep () from /lib64/libc.so.6
#1  0x00001486631d381e in sleep () from /lib64/libc.so.6
#2  0x000014865cd15cc0 in sig_pause_for_stacktrace () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02825/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-02-21-2300/lib/el8_amd64_gcc12/pluginFWCoreServicesPlugins.so
#3  <signal handler called>
#4  0x00001486631139bd in syscall () from /lib64/libc.so.6
#5  0x000014866436adf2 in tbb::detail::r1::futex_wait (comparand=2, futex=0x14865e26d0a4) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_14_0_0_pre2_SKYLAKEAVX512-el8_amd64_gcc12/build/CMSSW_14_0_0_pre2_SKYLAKEAVX512-build/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-5109e373398eaf79a8268d6a86a2f6e2/tbb-v2021.9.0/src/tbb/semaphore.h:100
#6  tbb::detail::r1::binary_semaphore::P (this=0x14865e26d0a4) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_14_0_0_pre2_SKYLAKEAVX512-el8_amd64_gcc12/build/CMSSW_14_0_0_pre2_SKYLAKEAVX512-build/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-5109e373398eaf79a8268d6a86a2f6e2/tbb-v2021.9.0/src/tbb/semaphore.h:253
#7  tbb::detail::r1::rml::internal::thread_monitor::wait (this=0x14865e26d0a0) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_14_0_0_pre2_SKYLAKEAVX512-el8_amd64_gcc12/build/CMSSW_14_0_0_pre2_SKYLAKEAVX512-build/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-5109e373398eaf79a8268d6a86a2f6e2/tbb-v2021.9.0/src/tbb/rml_thread_monitor.h:235
#8  tbb::detail::r1::rml::private_worker::run (this=0x14865e26d080) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_14_0_0_pre2_SKYLAKEAVX512-el8_amd64_gcc12/build/CMSSW_14_0_0_pre2_SKYLAKEAVX512-build/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-5109e373398eaf79a8268d6a86a2f6e2/tbb-v2021.9.0/src/tbb/private_server.cpp:273
#9  tbb::detail::r1::rml::private_worker::thread_routine (arg=0x14865e26d080) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_14_0_0_pre2_SKYLAKEAVX512-el8_amd64_gcc12/build/CMSSW_14_0_0_pre2_SKYLAKEAVX512-build/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-5109e373398eaf79a8268d6a86a2f6e2/tbb-v2021.9.0/src/tbb/private_server.cpp:221
#10 0x00001486634a71ca in start_thread () from /lib64/libpthread.so.0
#11 0x0000148663113e73 in clone () from /lib64/libc.so.6
Thread 2 (Thread 0x14863a3ff700 (LWP 2694273) "cmsRun"):
#0  0x00001486634b1672 in waitpid () from /lib64/libpthread.so.0
#1  0x000014865cd15e17 in edm::service::cmssw_stacktrace_fork() () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02825/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-02-21-2300/lib/el8_amd64_gcc12/pluginFWCoreServicesPlugins.so
#2  0x000014865cd18cea in edm::service::InitRootHandlers::stacktraceHelperThread() () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02825/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-02-21-2300/lib/el8_amd64_gcc12/pluginFWCoreServicesPlugins.so
#3  0x0000148663b3ba73 in std::execute_native_thread_routine (__p=0x14863d4504e0) at ../../../../../libstdc++-v3/src/c++11/thread.cc:82
#4  0x00001486634a71ca in start_thread () from /lib64/libpthread.so.0
#5  0x0000148663113e73 in clone () from /lib64/libc.so.6
Thread 1 (Thread 0x148662631640 (LWP 2692422) "cmsRun"):
#0  0x00001486631fe301 in poll () from /lib64/libc.so.6
#1  0x000014865cd646af in full_read.constprop () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02825/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-02-21-2300/lib/el8_amd64_gcc12/pluginFWCoreServicesPlugins.so
#2  0x000014865cd18dbc in edm::service::InitRootHandlers::stacktraceFromThread() () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02825/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-02-21-2300/lib/el8_amd64_gcc12/pluginFWCoreServicesPlugins.so
#3  0x000014865cd19720 in sig_dostack_then_abort () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02825/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-02-21-2300/lib/el8_amd64_gcc12/pluginFWCoreServicesPlugins.so
#4  <signal handler called>
#5  tbb::detail::r1::spawn (t=..., ctx=...) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_14_0_0_pre2_SKYLAKEAVX512-el8_amd64_gcc12/build/CMSSW_14_0_0_pre2_SKYLAKEAVX512-build/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-5109e373398eaf79a8268d6a86a2f6e2/tbb-v2021.9.0/src/tbb/task_dispatcher.cpp:36
#6  0x0000148665b20928 in tbb::detail::d1::function_task<edm::WaitingTaskHolder::doneWaiting(std::__exception_ptr::exception_ptr)::{lambda()#1}>::execute(tbb::detail::d1::execution_data&) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02825/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-02-21-2300/lib/el8_amd64_gcc12/libFWCoreFramework.so
#7  0x0000148664371281 in tbb::detail::r1::task_dispatcher::local_wait_for_all<false, tbb::detail::r1::external_waiter> (waiter=..., t=<optimized out>, this=0x148661b2b900) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_14_0_0_pre2_SKYLAKEAVX512-el8_amd64_gcc12/build/CMSSW_14_0_0_pre2_SKYLAKEAVX512-build/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-5109e373398eaf79a8268d6a86a2f6e2/tbb-v2021.9.0/src/tbb/task_dispatcher.h:322
#8  tbb::detail::r1::task_dispatcher::local_wait_for_all<tbb::detail::r1::external_waiter> (waiter=..., t=<optimized out>, this=0x148661b2b900) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_14_0_0_pre2_SKYLAKEAVX512-el8_amd64_gcc12/build/CMSSW_14_0_0_pre2_SKYLAKEAVX512-build/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-5109e373398eaf79a8268d6a86a2f6e2/tbb-v2021.9.0/src/tbb/task_dispatcher.h:458
#9  tbb::detail::r1::task_dispatcher::execute_and_wait (t=<optimized out>, wait_ctx=..., w_ctx=...) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_14_0_0_pre2_SKYLAKEAVX512-el8_amd64_gcc12/build/CMSSW_14_0_0_pre2_SKYLAKEAVX512-build/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-5109e373398eaf79a8268d6a86a2f6e2/tbb-v2021.9.0/src/tbb/task_dispatcher.cpp:168
#10 0x0000148665b30cdb in edm::FinalWaitingTask::wait() () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02825/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-02-21-2300/lib/el8_amd64_gcc12/libFWCoreFramework.so
#11 0x0000148665b40428 in edm::EventProcessor::endUnfinishedRun(bool) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02825/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-02-21-2300/lib/el8_amd64_gcc12/libFWCoreFramework.so
#12 0x0000148665b34f63 in edm::(anonymous namespace)::RunResources::~RunResources() () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02825/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-02-21-2300/lib/el8_amd64_gcc12/libFWCoreFramework.so
#13 0x0000148665b3aaf6 in edm::EventProcessor::runToCompletion() () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02825/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-02-21-2300/lib/el8_amd64_gcc12/libFWCoreFramework.so
#14 0x00000000004074ef in tbb::detail::d1::task_arena_function<main::{lambda()#1}::operator()() const::{lambda()#1}, void>::operator()() const ()
#15 0x000014866435d9ad in tbb::detail::r1::task_arena_impl::execute (ta=..., d=...) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_14_0_0_pre2_SKYLAKEAVX512-el8_amd64_gcc12/build/CMSSW_14_0_0_pre2_SKYLAKEAVX512-build/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-5109e373398eaf79a8268d6a86a2f6e2/tbb-v2021.9.0/src/tbb/arena.cpp:688
#16 0x0000000000408ed2 in main::{lambda()#1}::operator()() const ()
#17 0x000000000040517c in main ()

Current Modules:

Module: none (crashed)/data/cmsbld/jenkins/workspace/ib-run-qa/CMSSW_14_1_X_2024-02-21-2300/src/DQM/Integration/test/runtest.sh: line 28: 2692422 Segmentation fault      (core dumped) cmsRun $CLIENTS_DIR/$1 unitTest=True

---> test TestDQMOnlineClient-visualization had ERRORS

@aandvalenzuela
Copy link
Contributor

Hello, new segfault occurrence in CMSSW_14_1_X_2024-07-17-2300:

%MSG
Begin processing the 98th record. Run 355380, Event 26705277, LumiSection 19 on stream 4 at 18-Jul-2024 02:49:28.143 CEST
Begin processing the 99th record. Run 355380, Event 26732456, LumiSection 19 on stream 1 at 18-Jul-2024 02:49:28.672 CEST
Begin processing the 100th record. Run 355380, Event 26785259, LumiSection 19 on stream 0 at 18-Jul-2024 02:49:29.123 CEST


A fatal system signal has occurred: segmentation violation
The following is the call stack containing the origin of the signal.

Thu Jul 18 02:49:36 CEST 2024
Thread 26 (Thread 0x1506d83ff700 (LWP 1304338) "cmsRun"):
#0  0x000015075f66c48c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00001507264dfba6 in Eigen::ThreadPoolTempl<tsl::thread::EigenEnvironment>::WaitForWork(Eigen::EventCount::Waiter*, tsl::thread::EigenEnvironment::Task*) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02846/el8_amd64_gcc12/cms/cmssw-patch/CMSSW_14_1_X_2024-07-17-2300/external/el8_amd64_gcc12/lib/libtensorflow_cc.so.2
#2  0x00001507264e01a5 in Eigen::ThreadPoolTempl<tsl::thread::EigenEnvironment>::WorkerLoop(int) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02846/el8_amd64_gcc12/cms/cmssw-patch/CMSSW_14_1_X_2024-07-17-2300/external/el8_amd64_gcc12/lib/libtensorflow_cc.so.2
#3  0x00001507264dd968 in std::_Function_handler<void (), tsl::thread::EigenEnvironment::CreateThread(std::function<void ()>)::{lambda()#1}>::_M_invoke(std::_Any_data const&) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02846/el8_amd64_gcc12/cms/cmssw-patch/CMSSW_14_1_X_2024-07-17-2300/external/el8_amd64_gcc12/lib/libtensorflow_cc.so.2
#4  0x0000150715878472 in tsl::(anonymous namespace)::PThread::ThreadFn(void*) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02846/el8_amd64_gcc12/cms/cmssw-patch/CMSSW_14_1_X_2024-07-17-2300/external/el8_amd64_gcc12/lib/libtensorflow_framework.so.2
#5  0x000015075f6661ca in start_thread () from /lib64/libpthread.so.0
#6  0x000015075f2c18d3 in clone () from /lib64/libc.so.6
Thread 25 (Thread 0x1506d8dff700 (LWP 1304337) "cmsRun"):
#0  0x000015075f66c48c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00001507264dfba6 in Eigen::ThreadPoolTempl<tsl::thread::EigenEnvironment>::WaitForWork(Eigen::EventCount::Waiter*, tsl::thread::EigenEnvironment::Task*) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02846/el8_amd64_gcc12/cms/cmssw-patch/CMSSW_14_1_X_2024-07-17-2300/external/el8_amd64_gcc12/lib/libtensorflow_cc.so.2
#2  0x00001507264e01a5 in Eigen::ThreadPoolTempl<tsl::thread::EigenEnvironment>::WorkerLoop(int) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02846/el8_amd64_gcc12/cms/cmssw-patch/CMSSW_14_1_X_2024-07-17-2300/external/el8_amd64_gcc12/lib/libtensorflow_cc.so.2
#3  0x00001507264dd968 in std::_Function_handler<void (), tsl::thread::EigenEnvironment::CreateThread(std::function<void ()>)::{lambda()#1}>::_M_invoke(std::_Any_data const&) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02846/el8_amd64_gcc12/cms/cmssw-patch/CMSSW_14_1_X_2024-07-17-2300/external/el8_amd64_gcc12/lib/libtensorflow_cc.so.2
#4  0x0000150715878472 in tsl::(anonymous namespace)::PThread::ThreadFn(void*) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02846/el8_amd64_gcc12/cms/cmssw-patch/CMSSW_14_1_X_2024-07-17-2300/external/el8_amd64_gcc12/lib/libtensorflow_framework.so.2
#5  0x000015075f6661ca in start_thread () from /lib64/libpthread.so.0
#6  0x000015075f2c18d3 in clone () from /lib64/libc.so.6
Thread 24 (Thread 0x1506d92de700 (LWP 1304336) "cmsRun"):
#0  0x000015075f66c48c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00001507264dfba6 in Eigen::ThreadPoolTempl<tsl::thread::EigenEnvironment>::WaitForWork(Eigen::EventCount::Waiter*, tsl::thread::EigenEnvironment::Task*) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02846/el8_amd64_gcc12/cms/cmssw-patch/CMSSW_14_1_X_2024-07-17-2300/external/el8_amd64_gcc12/lib/libtensorflow_cc.so.2
#2  0x00001507264e01a5 in Eigen::ThreadPoolTempl<tsl::thread::EigenEnvironment>::WorkerLoop(int) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02846/el8_amd64_gcc12/cms/cmssw-patch/CMSSW_14_1_X_2024-07-17-2300/external/el8_amd64_gcc12/lib/libtensorflow_cc.so.2
#3  0x00001507264dd968 in std::_Function_handler<void (), tsl::thread::EigenEnvironment::CreateThread(std::function<void ()>)::{lambda()#1}>::_M_invoke(std::_Any_data const&) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02846/el8_amd64_gcc12/cms/cmssw-patch/CMSSW_14_1_X_2024-07-17-2300/external/el8_amd64_gcc12/lib/libtensorflow_cc.so.2
#4  0x0000150715878472 in tsl::(anonymous namespace)::PThread::ThreadFn(void*) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02846/el8_amd64_gcc12/cms/cmssw-patch/CMSSW_14_1_X_2024-07-17-2300/external/el8_amd64_gcc12/lib/libtensorflow_framework.so.2
#5  0x000015075f6661ca in start_thread () from /lib64/libpthread.so.0
#6  0x000015075f2c18d3 in clone () from /lib64/libc.so.6
Thread 23 (Thread 0x150705bf1700 (LWP 1304251) "cmsRun"):
#0  0x000015075f66edd6 in do_futex_wait.constprop () from /lib64/libpthread.so.0
#1  0x000015075f66eec8 in __new_sem_wait_slow.constprop.0 () from /lib64/libpthread.so.0
#2  0x000015075779d706 in XrdCl::JobManager::RunJobs() () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02846/el8_amd64_gcc12/cms/cmssw-patch/CMSSW_14_1_X_2024-07-17-2300/external/el8_amd64_gcc12/lib/libXrdCl.so.3
#3  0x000015075779d7b9 in RunRunnerThread () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02846/el8_amd64_gcc12/cms/cmssw-patch/CMSSW_14_1_X_2024-07-17-2300/external/el8_amd64_gcc12/lib/libXrdCl.so.3
#4  0x000015075f6661ca in start_thread () from /lib64/libpthread.so.0
#5  0x000015075f2c18d3 in clone () from /lib64/libc.so.6
Thread 22 (Thread 0x150705df2700 (LWP 1304250) "cmsRun"):
#0  0x000015075f66edd6 in do_futex_wait.constprop () from /lib64/libpthread.so.0
#1  0x000015075f66eec8 in __new_sem_wait_slow.constprop.0 () from /lib64/libpthread.so.0
#2  0x000015075779d706 in XrdCl::JobManager::RunJobs() () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02846/el8_amd64_gcc12/cms/cmssw-patch/CMSSW_14_1_X_2024-07-17-2300/external/el8_amd64_gcc12/lib/libXrdCl.so.3
#3  0x000015075779d7b9 in RunRunnerThread () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02846/el8_amd64_gcc12/cms/cmssw-patch/CMSSW_14_1_X_2024-07-17-2300/external/el8_amd64_gcc12/lib/libXrdCl.so.3
#4  0x000015075f6661ca in start_thread () from /lib64/libpthread.so.0
#5  0x000015075f2c18d3 in clone () from /lib64/libc.so.6
Thread 21 (Thread 0x150705ff3700 (LWP 1304249) "cmsRun"):
#0  0x000015075f66edd6 in do_futex_wait.constprop () from /lib64/libpthread.so.0
#1  0x000015075f66eec8 in __new_sem_wait_slow.constprop.0 () from /lib64/libpthread.so.0
#2  0x000015075779d706 in XrdCl::JobManager::RunJobs() () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02846/el8_amd64_gcc12/cms/cmssw-patch/CMSSW_14_1_X_2024-07-17-2300/external/el8_amd64_gcc12/lib/libXrdCl.so.3
#3  0x000015075779d7b9 in RunRunnerThread () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02846/el8_amd64_gcc12/cms/cmssw-patch/CMSSW_14_1_X_2024-07-17-2300/external/el8_amd64_gcc12/lib/libXrdCl.so.3
#4  0x000015075f6661ca in start_thread () from /lib64/libpthread.so.0
#5  0x000015075f2c18d3 in clone () from /lib64/libc.so.6
Thread 20 (Thread 0x1507061f4700 (LWP 1304248) "cmsRun"):
#0  0x000015075f6701b0 in nanosleep () from /lib64/libpthread.so.0
#1  0x0000150759019f08 in XrdSysTimer::Wait(int) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02846/el8_amd64_gcc12/cms/cmssw-patch/CMSSW_14_1_X_2024-07-17-2300/external/el8_amd64_gcc12/lib/libXrdUtils.so.3
#2  0x000015075771203c in XrdCl::TaskManager::RunTasks() () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02846/el8_amd64_gcc12/cms/cmssw-patch/CMSSW_14_1_X_2024-07-17-2300/external/el8_amd64_gcc12/lib/libXrdCl.so.3
#3  0x0000150757712179 in RunRunnerThread () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02846/el8_amd64_gcc12/cms/cmssw-patch/CMSSW_14_1_X_2024-07-17-2300/external/el8_amd64_gcc12/lib/libXrdCl.so.3
#4  0x000015075f6661ca in start_thread () from /lib64/libpthread.so.0
#5  0x000015075f2c18d3 in clone () from /lib64/libc.so.6
Thread 19 (Thread 0x1507063f5700 (LWP 1304247) "cmsRun"):
#0  0x000015075f3c7307 in epoll_wait () from /lib64/libc.so.6
#1  0x0000150759014282 in XrdSys::IOEvents::PollE::Begin(XrdSysSemaphore*, int&, char const**) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02846/el8_amd64_gcc12/cms/cmssw-patch/CMSSW_14_1_X_2024-07-17-2300/external/el8_amd64_gcc12/lib/libXrdUtils.so.3
#2  0x00001507590101ed in XrdSys::IOEvents::BootStrap::Start(void*) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02846/el8_amd64_gcc12/cms/cmssw-patch/CMSSW_14_1_X_2024-07-17-2300/external/el8_amd64_gcc12/lib/libXrdUtils.so.3
#3  0x0000150759019617 in XrdSysThread_Xeq () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02846/el8_amd64_gcc12/cms/cmssw-patch/CMSSW_14_1_X_2024-07-17-2300/external/el8_amd64_gcc12/lib/libXrdUtils.so.3
#4  0x000015075f6661ca in start_thread () from /lib64/libpthread.so.0
#5  0x000015075f2c18d3 in clone () from /lib64/libc.so.6
Thread 18 (Thread 0x1507065f6700 (LWP 1304246) "cmsRun"):
#0  0x000015075f3c7307 in epoll_wait () from /lib64/libc.so.6
#1  0x0000150759014282 in XrdSys::IOEvents::PollE::Begin(XrdSysSemaphore*, int&, char const**) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02846/el8_amd64_gcc12/cms/cmssw-patch/CMSSW_14_1_X_2024-07-17-2300/external/el8_amd64_gcc12/lib/libXrdUtils.so.3
#2  0x00001507590101ed in XrdSys::IOEvents::BootStrap::Start(void*) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02846/el8_amd64_gcc12/cms/cmssw-patch/CMSSW_14_1_X_2024-07-17-2300/external/el8_amd64_gcc12/lib/libXrdUtils.so.3
#3  0x0000150759019617 in XrdSysThread_Xeq () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02846/el8_amd64_gcc12/cms/cmssw-patch/CMSSW_14_1_X_2024-07-17-2300/external/el8_amd64_gcc12/lib/libXrdUtils.so.3
#4  0x000015075f6661ca in start_thread () from /lib64/libpthread.so.0
#5  0x000015075f2c18d3 in clone () from /lib64/libc.so.6
Thread 17 (Thread 0x1507067f7700 (LWP 1304245) "cmsRun"):
#0  0x000015075f3c7307 in epoll_wait () from /lib64/libc.so.6
#1  0x0000150759014282 in XrdSys::IOEvents::PollE::Begin(XrdSysSemaphore*, int&, char const**) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02846/el8_amd64_gcc12/cms/cmssw-patch/CMSSW_14_1_X_2024-07-17-2300/external/el8_amd64_gcc12/lib/libXrdUtils.so.3
#2  0x00001507590101ed in XrdSys::IOEvents::BootStrap::Start(void*) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02846/el8_amd64_gcc12/cms/cmssw-patch/CMSSW_14_1_X_2024-07-17-2300/external/el8_amd64_gcc12/lib/libXrdUtils.so.3
#3  0x0000150759019617 in XrdSysThread_Xeq () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02846/el8_amd64_gcc12/cms/cmssw-patch/CMSSW_14_1_X_2024-07-17-2300/external/el8_amd64_gcc12/lib/libXrdUtils.so.3
#4  0x000015075f6661ca in start_thread () from /lib64/libpthread.so.0
#5  0x000015075f2c18d3 in clone () from /lib64/libc.so.6
Thread 16 (Thread 0x1507069f8700 (LWP 1304244) "cmsRun"):
#0  0x000015075f3c7307 in epoll_wait () from /lib64/libc.so.6
#1  0x0000150759014282 in XrdSys::IOEvents::PollE::Begin(XrdSysSemaphore*, int&, char const**) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02846/el8_amd64_gcc12/cms/cmssw-patch/CMSSW_14_1_X_2024-07-17-2300/external/el8_amd64_gcc12/lib/libXrdUtils.so.3
#2  0x00001507590101ed in XrdSys::IOEvents::BootStrap::Start(void*) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02846/el8_amd64_gcc12/cms/cmssw-patch/CMSSW_14_1_X_2024-07-17-2300/external/el8_amd64_gcc12/lib/libXrdUtils.so.3
#3  0x0000150759019617 in XrdSysThread_Xeq () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02846/el8_amd64_gcc12/cms/cmssw-patch/CMSSW_14_1_X_2024-07-17-2300/external/el8_amd64_gcc12/lib/libXrdUtils.so.3
#4  0x000015075f6661ca in start_thread () from /lib64/libpthread.so.0
#5  0x000015075f2c18d3 in clone () from /lib64/libc.so.6
Thread 15 (Thread 0x150706bf9700 (LWP 1304243) "cmsRun"):
#0  0x000015075f3c7307 in epoll_wait () from /lib64/libc.so.6
#1  0x0000150759014282 in XrdSys::IOEvents::PollE::Begin(XrdSysSemaphore*, int&, char const**) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02846/el8_amd64_gcc12/cms/cmssw-patch/CMSSW_14_1_X_2024-07-17-2300/external/el8_amd64_gcc12/lib/libXrdUtils.so.3
#2  0x00001507590101ed in XrdSys::IOEvents::BootStrap::Start(void*) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02846/el8_amd64_gcc12/cms/cmssw-patch/CMSSW_14_1_X_2024-07-17-2300/external/el8_amd64_gcc12/lib/libXrdUtils.so.3
#3  0x0000150759019617 in XrdSysThread_Xeq () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02846/el8_amd64_gcc12/cms/cmssw-patch/CMSSW_14_1_X_2024-07-17-2300/external/el8_amd64_gcc12/lib/libXrdUtils.so.3
#4  0x000015075f6661ca in start_thread () from /lib64/libpthread.so.0
#5  0x000015075f2c18d3 in clone () from /lib64/libc.so.6
Thread 14 (Thread 0x150706dfa700 (LWP 1304242) "cmsRun"):
#0  0x000015075f3c7307 in epoll_wait () from /lib64/libc.so.6
#1  0x0000150759014282 in XrdSys::IOEvents::PollE::Begin(XrdSysSemaphore*, int&, char const**) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02846/el8_amd64_gcc12/cms/cmssw-patch/CMSSW_14_1_X_2024-07-17-2300/external/el8_amd64_gcc12/lib/libXrdUtils.so.3
#2  0x00001507590101ed in XrdSys::IOEvents::BootStrap::Start(void*) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02846/el8_amd64_gcc12/cms/cmssw-patch/CMSSW_14_1_X_2024-07-17-2300/external/el8_amd64_gcc12/lib/libXrdUtils.so.3
#3  0x0000150759019617 in XrdSysThread_Xeq () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02846/el8_amd64_gcc12/cms/cmssw-patch/CMSSW_14_1_X_2024-07-17-2300/external/el8_amd64_gcc12/lib/libXrdUtils.so.3
#4  0x000015075f6661ca in start_thread () from /lib64/libpthread.so.0
#5  0x000015075f2c18d3 in clone () from /lib64/libc.so.6
Thread 13 (Thread 0x150706ffb700 (LWP 1304241) "cmsRun"):
#0  0x000015075f3c7307 in epoll_wait () from /lib64/libc.so.6
#1  0x0000150759014282 in XrdSys::IOEvents::PollE::Begin(XrdSysSemaphore*, int&, char const**) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02846/el8_amd64_gcc12/cms/cmssw-patch/CMSSW_14_1_X_2024-07-17-2300/external/el8_amd64_gcc12/lib/libXrdUtils.so.3
#2  0x00001507590101ed in XrdSys::IOEvents::BootStrap::Start(void*) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02846/el8_amd64_gcc12/cms/cmssw-patch/CMSSW_14_1_X_2024-07-17-2300/external/el8_amd64_gcc12/lib/libXrdUtils.so.3
#3  0x0000150759019617 in XrdSysThread_Xeq () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02846/el8_amd64_gcc12/cms/cmssw-patch/CMSSW_14_1_X_2024-07-17-2300/external/el8_amd64_gcc12/lib/libXrdUtils.so.3
#4  0x000015075f6661ca in start_thread () from /lib64/libpthread.so.0
#5  0x000015075f2c18d3 in clone () from /lib64/libc.so.6
Thread 12 (Thread 0x1507071fc700 (LWP 1304240) "cmsRun"):
#0  0x000015075f3c7307 in epoll_wait () from /lib64/libc.so.6
#1  0x0000150759014282 in XrdSys::IOEvents::PollE::Begin(XrdSysSemaphore*, int&, char const**) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02846/el8_amd64_gcc12/cms/cmssw-patch/CMSSW_14_1_X_2024-07-17-2300/external/el8_amd64_gcc12/lib/libXrdUtils.so.3
#2  0x00001507590101ed in XrdSys::IOEvents::BootStrap::Start(void*) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02846/el8_amd64_gcc12/cms/cmssw-patch/CMSSW_14_1_X_2024-07-17-2300/external/el8_amd64_gcc12/lib/libXrdUtils.so.3
#3  0x0000150759019617 in XrdSysThread_Xeq () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02846/el8_amd64_gcc12/cms/cmssw-patch/CMSSW_14_1_X_2024-07-17-2300/external/el8_amd64_gcc12/lib/libXrdUtils.so.3
#4  0x000015075f6661ca in start_thread () from /lib64/libpthread.so.0
#5  0x000015075f2c18d3 in clone () from /lib64/libc.so.6
Thread 11 (Thread 0x1507073fd700 (LWP 1304239) "cmsRun"):
#0  0x000015075f3c7307 in epoll_wait () from /lib64/libc.so.6
#1  0x0000150759014282 in XrdSys::IOEvents::PollE::Begin(XrdSysSemaphore*, int&, char const**) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02846/el8_amd64_gcc12/cms/cmssw-patch/CMSSW_14_1_X_2024-07-17-2300/external/el8_amd64_gcc12/lib/libXrdUtils.so.3
#2  0x00001507590101ed in XrdSys::IOEvents::BootStrap::Start(void*) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02846/el8_amd64_gcc12/cms/cmssw-patch/CMSSW_14_1_X_2024-07-17-2300/external/el8_amd64_gcc12/lib/libXrdUtils.so.3
#3  0x0000150759019617 in XrdSysThread_Xeq () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02846/el8_amd64_gcc12/cms/cmssw-patch/CMSSW_14_1_X_2024-07-17-2300/external/el8_amd64_gcc12/lib/libXrdUtils.so.3
#4  0x000015075f6661ca in start_thread () from /lib64/libpthread.so.0
#5  0x000015075f2c18d3 in clone () from /lib64/libc.so.6
Thread 10 (Thread 0x1507075fe700 (LWP 1304238) "cmsRun"):
#0  0x000015075f3c7307 in epoll_wait () from /lib64/libc.so.6
#1  0x0000150759014282 in XrdSys::IOEvents::PollE::Begin(XrdSysSemaphore*, int&, char const**) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02846/el8_amd64_gcc12/cms/cmssw-patch/CMSSW_14_1_X_2024-07-17-2300/external/el8_amd64_gcc12/lib/libXrdUtils.so.3
#2  0x00001507590101ed in XrdSys::IOEvents::BootStrap::Start(void*) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02846/el8_amd64_gcc12/cms/cmssw-patch/CMSSW_14_1_X_2024-07-17-2300/external/el8_amd64_gcc12/lib/libXrdUtils.so.3
#3  0x0000150759019617 in XrdSysThread_Xeq () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02846/el8_amd64_gcc12/cms/cmssw-patch/CMSSW_14_1_X_2024-07-17-2300/external/el8_amd64_gcc12/lib/libXrdUtils.so.3
#4  0x000015075f6661ca in start_thread () from /lib64/libpthread.so.0
#5  0x000015075f2c18d3 in clone () from /lib64/libc.so.6
Thread 9 (Thread 0x1507081ff700 (LWP 1304237) "cmsRun"):
#0  0x000015075f390098 in nanosleep () from /lib64/libc.so.6
#1  0x000015075f38ff9e in sleep () from /lib64/libc.so.6
#2  0x00001507599dfa10 in sig_pause_for_stacktrace () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02846/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-07-17-1200/lib/el8_amd64_gcc12/pluginFWCoreServicesPlugins.so
#3  <signal handler called>
#4  0x000015075f2c141d in syscall () from /lib64/libc.so.6
#5  0x00001507613d3fd2 in tbb::detail::r1::futex_wait (comparand=2, futex=0x15075a0d4e24) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-df2a8868ba5d9111864e1a63085d7a4b/tbb-v2021.9.0/src/tbb/semaphore.h:100
#6  tbb::detail::r1::binary_semaphore::P (this=0x15075a0d4e24) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-df2a8868ba5d9111864e1a63085d7a4b/tbb-v2021.9.0/src/tbb/semaphore.h:253
#7  tbb::detail::r1::rml::internal::thread_monitor::wait (this=0x15075a0d4e20) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-df2a8868ba5d9111864e1a63085d7a4b/tbb-v2021.9.0/src/tbb/rml_thread_monitor.h:235
#8  tbb::detail::r1::rml::private_worker::run (this=0x15075a0d4e00) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-df2a8868ba5d9111864e1a63085d7a4b/tbb-v2021.9.0/src/tbb/private_server.cpp:273
#9  tbb::detail::r1::rml::private_worker::thread_routine (arg=0x15075a0d4e00) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-df2a8868ba5d9111864e1a63085d7a4b/tbb-v2021.9.0/src/tbb/private_server.cpp:221
#10 0x000015075f6661ca in start_thread () from /lib64/libpthread.so.0
#11 0x000015075f2c18d3 in clone () from /lib64/libc.so.6
Thread 8 (Thread 0x1507095fd700 (LWP 1304236) "cmsRun"):
#0  0x000015075f390098 in nanosleep () from /lib64/libc.so.6
#1  0x000015075f38ff9e in sleep () from /lib64/libc.so.6
#2  0x00001507599dfa10 in sig_pause_for_stacktrace () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02846/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-07-17-1200/lib/el8_amd64_gcc12/pluginFWCoreServicesPlugins.so
#3  <signal handler called>
#4  0x000015075f2c141d in syscall () from /lib64/libc.so.6
#5  0x00001507613d3fd2 in tbb::detail::r1::futex_wait (comparand=2, futex=0x15075a0d4f24) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-df2a8868ba5d9111864e1a63085d7a4b/tbb-v2021.9.0/src/tbb/semaphore.h:100
#6  tbb::detail::r1::binary_semaphore::P (this=0x15075a0d4f24) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-df2a8868ba5d9111864e1a63085d7a4b/tbb-v2021.9.0/src/tbb/semaphore.h:253
#7  tbb::detail::r1::rml::internal::thread_monitor::wait (this=0x15075a0d4f20) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-df2a8868ba5d9111864e1a63085d7a4b/tbb-v2021.9.0/src/tbb/rml_thread_monitor.h:235
#8  tbb::detail::r1::rml::private_worker::run (this=0x15075a0d4f00) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-df2a8868ba5d9111864e1a63085d7a4b/tbb-v2021.9.0/src/tbb/private_server.cpp:273
#9  tbb::detail::r1::rml::private_worker::thread_routine (arg=0x15075a0d4f00) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-df2a8868ba5d9111864e1a63085d7a4b/tbb-v2021.9.0/src/tbb/private_server.cpp:221
#10 0x000015075f6661ca in start_thread () from /lib64/libpthread.so.0
#11 0x000015075f2c18d3 in clone () from /lib64/libc.so.6
Thread 7 (Thread 0x150709ffe700 (LWP 1304235) "cmsRun"):
#0  0x000015075f390098 in nanosleep () from /lib64/libc.so.6
#1  0x000015075f38ff9e in sleep () from /lib64/libc.so.6
#2  0x00001507599dfa10 in sig_pause_for_stacktrace () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02846/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-07-17-1200/lib/el8_amd64_gcc12/pluginFWCoreServicesPlugins.so
#3  <signal handler called>
#4  0x000015075f2c141d in syscall () from /lib64/libc.so.6
#5  0x00001507613d3fd2 in tbb::detail::r1::futex_wait (comparand=2, futex=0x15075a0d5024) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-df2a8868ba5d9111864e1a63085d7a4b/tbb-v2021.9.0/src/tbb/semaphore.h:100
#6  tbb::detail::r1::binary_semaphore::P (this=0x15075a0d5024) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-df2a8868ba5d9111864e1a63085d7a4b/tbb-v2021.9.0/src/tbb/semaphore.h:253
#7  tbb::detail::r1::rml::internal::thread_monitor::wait (this=0x15075a0d5020) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-df2a8868ba5d9111864e1a63085d7a4b/tbb-v2021.9.0/src/tbb/rml_thread_monitor.h:235
#8  tbb::detail::r1::rml::private_worker::run (this=0x15075a0d5000) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-df2a8868ba5d9111864e1a63085d7a4b/tbb-v2021.9.0/src/tbb/private_server.cpp:273
#9  tbb::detail::r1::rml::private_worker::thread_routine (arg=0x15075a0d5000) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-df2a8868ba5d9111864e1a63085d7a4b/tbb-v2021.9.0/src/tbb/private_server.cpp:221
#10 0x000015075f6661ca in start_thread () from /lib64/libpthread.so.0
#11 0x000015075f2c18d3 in clone () from /lib64/libc.so.6
Thread 6 (Thread 0x15070a9ff700 (LWP 1304234) "cmsRun"):
#0  0x000015075f390098 in nanosleep () from /lib64/libc.so.6
#1  0x000015075f38ff9e in sleep () from /lib64/libc.so.6
#2  0x00001507599dfa10 in sig_pause_for_stacktrace () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02846/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-07-17-1200/lib/el8_amd64_gcc12/pluginFWCoreServicesPlugins.so
#3  <signal handler called>
#4  0x000015075f2c141d in syscall () from /lib64/libc.so.6
#5  0x00001507613d3fd2 in tbb::detail::r1::futex_wait (comparand=2, futex=0x15075a0d4ea4) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-df2a8868ba5d9111864e1a63085d7a4b/tbb-v2021.9.0/src/tbb/semaphore.h:100
#6  tbb::detail::r1::binary_semaphore::P (this=0x15075a0d4ea4) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-df2a8868ba5d9111864e1a63085d7a4b/tbb-v2021.9.0/src/tbb/semaphore.h:253
#7  tbb::detail::r1::rml::internal::thread_monitor::wait (this=0x15075a0d4ea0) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-df2a8868ba5d9111864e1a63085d7a4b/tbb-v2021.9.0/src/tbb/rml_thread_monitor.h:235
#8  tbb::detail::r1::rml::private_worker::run (this=0x15075a0d4e80) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-df2a8868ba5d9111864e1a63085d7a4b/tbb-v2021.9.0/src/tbb/private_server.cpp:273
#9  tbb::detail::r1::rml::private_worker::thread_routine (arg=0x15075a0d4e80) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-df2a8868ba5d9111864e1a63085d7a4b/tbb-v2021.9.0/src/tbb/private_server.cpp:221
#10 0x000015075f6661ca in start_thread () from /lib64/libpthread.so.0
#11 0x000015075f2c18d3 in clone () from /lib64/libc.so.6
Thread 5 (Thread 0x15070b76b700 (LWP 1304233) "cmsRun"):
#0  0x000015075f390098 in nanosleep () from /lib64/libc.so.6
#1  0x000015075f38ff9e in sleep () from /lib64/libc.so.6
#2  0x00001507599dfa10 in sig_pause_for_stacktrace () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02846/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-07-17-1200/lib/el8_amd64_gcc12/pluginFWCoreServicesPlugins.so
#3  <signal handler called>
#4  0x000015075f2c141d in syscall () from /lib64/libc.so.6
#5  0x00001507613d3fd2 in tbb::detail::r1::futex_wait (comparand=2, futex=0x15075a0d4fa4) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-df2a8868ba5d9111864e1a63085d7a4b/tbb-v2021.9.0/src/tbb/semaphore.h:100
#6  tbb::detail::r1::binary_semaphore::P (this=0x15075a0d4fa4) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-df2a8868ba5d9111864e1a63085d7a4b/tbb-v2021.9.0/src/tbb/semaphore.h:253
#7  tbb::detail::r1::rml::internal::thread_monitor::wait (this=0x15075a0d4fa0) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-df2a8868ba5d9111864e1a63085d7a4b/tbb-v2021.9.0/src/tbb/rml_thread_monitor.h:235
#8  tbb::detail::r1::rml::private_worker::run (this=0x15075a0d4f80) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-df2a8868ba5d9111864e1a63085d7a4b/tbb-v2021.9.0/src/tbb/private_server.cpp:273
#9  tbb::detail::r1::rml::private_worker::thread_routine (arg=0x15075a0d4f80) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-df2a8868ba5d9111864e1a63085d7a4b/tbb-v2021.9.0/src/tbb/private_server.cpp:221
#10 0x000015075f6661ca in start_thread () from /lib64/libpthread.so.0
#11 0x000015075f2c18d3 in clone () from /lib64/libc.so.6
Thread 4 (Thread 0x15070c890700 (LWP 1304232) "cmsRun"):
#0  0x000015075f390098 in nanosleep () from /lib64/libc.so.6
#1  0x000015075f38ff9e in sleep () from /lib64/libc.so.6
#2  0x00001507599dfa10 in sig_pause_for_stacktrace () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02846/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-07-17-1200/lib/el8_amd64_gcc12/pluginFWCoreServicesPlugins.so
#3  <signal handler called>
#4  0x000015075f2c140d in syscall () from /lib64/libc.so.6
#5  0x00001507613d346b in tbb::detail::r1::futex_wakeup_one (futex=0x15075a0d4fa4) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-df2a8868ba5d9111864e1a63085d7a4b/tbb-v2021.9.0/src/tbb/semaphore.h:109
#6  tbb::detail::r1::binary_semaphore::V (this=0x15075a0d4fa4) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-df2a8868ba5d9111864e1a63085d7a4b/tbb-v2021.9.0/src/tbb/semaphore.h:262
#7  tbb::detail::r1::rml::internal::thread_monitor::notify (this=0x15075a0d4fa0) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-df2a8868ba5d9111864e1a63085d7a4b/tbb-v2021.9.0/src/tbb/rml_thread_monitor.h:230
#8  tbb::detail::r1::rml::private_worker::wake_or_launch (this=0x15075a0d4f80) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-df2a8868ba5d9111864e1a63085d7a4b/tbb-v2021.9.0/src/tbb/private_server.cpp:292
#9  tbb::detail::r1::rml::private_server::wake_some (this=<optimized out>, additional_slack=<optimized out>) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-df2a8868ba5d9111864e1a63085d7a4b/tbb-v2021.9.0/src/tbb/private_server.cpp:412
#10 0x00001507613d3e49 in tbb::detail::r1::rml::private_server::propagate_chain_reaction (this=<optimized out>) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-df2a8868ba5d9111864e1a63085d7a4b/tbb-v2021.9.0/src/tbb/private_server.cpp:159
#11 tbb::detail::r1::rml::private_worker::run (this=0x15075a0d5100) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-df2a8868ba5d9111864e1a63085d7a4b/tbb-v2021.9.0/src/tbb/private_server.cpp:275
#12 tbb::detail::r1::rml::private_worker::thread_routine (arg=0x15075a0d5100) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-df2a8868ba5d9111864e1a63085d7a4b/tbb-v2021.9.0/src/tbb/private_server.cpp:221
#13 0x000015075f6661ca in start_thread () from /lib64/libpthread.so.0
#14 0x000015075f2c18d3 in clone () from /lib64/libc.so.6
Thread 3 (Thread 0x15070d291700 (LWP 1304231) "cmsRun"):
#0  0x000015075f390098 in nanosleep () from /lib64/libc.so.6
#1  0x000015075f38ff9e in sleep () from /lib64/libc.so.6
#2  0x00001507599dfa10 in sig_pause_for_stacktrace () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02846/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-07-17-1200/lib/el8_amd64_gcc12/pluginFWCoreServicesPlugins.so
#3  <signal handler called>
#4  0x0000150762032377 in edm::Worker::reset() () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02846/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-07-17-1200/lib/el8_amd64_gcc12/libFWCoreFramework.so
#5  0x0000150762032dc4 in edm::WorkerManager::resetAll() () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02846/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-07-17-1200/lib/el8_amd64_gcc12/libFWCoreFramework.so
#6  0x0000150761f5d10d in edm::StreamSchedule::processOneStreamAsync<edm::OccurrenceTraits<edm::RunPrincipal, (edm::BranchActionType)2> >(edm::WaitingTaskHolder, edm::OccurrenceTraits<edm::RunPrincipal, (edm::BranchActionType)2>::TransitionInfoType&, edm::ServiceToken const&, bool)::{lambda()#2}::operator()() () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02846/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-07-17-1200/lib/el8_amd64_gcc12/libFWCoreFramework.so
#7  0x0000150761f6dedd in tbb::detail::d1::function_task<edm::StreamSchedule::processOneStreamAsync<edm::OccurrenceTraits<edm::RunPrincipal, (edm::BranchActionType)2> >(edm::WaitingTaskHolder, edm::OccurrenceTraits<edm::RunPrincipal, (edm::BranchActionType)2>::TransitionInfoType&, edm::ServiceToken const&, bool)::{lambda()#3}>::execute(tbb::detail::d1::execution_data&) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02846/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-07-17-1200/lib/el8_amd64_gcc12/libFWCoreFramework.so
#8  0x00001507613d1b3b in tbb::detail::r1::task_dispatcher::local_wait_for_all<false, tbb::detail::r1::outermost_worker_waiter> (t=0x150708769100, waiter=..., this=0x15075ddbba00) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-df2a8868ba5d9111864e1a63085d7a4b/tbb-v2021.9.0/src/tbb/task_dispatcher.h:322
#9  tbb::detail::r1::task_dispatcher::local_wait_for_all<tbb::detail::r1::outermost_worker_waiter> (t=0x0, waiter=..., this=0x15075ddbba00) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-df2a8868ba5d9111864e1a63085d7a4b/tbb-v2021.9.0/src/tbb/task_dispatcher.h:458
#10 tbb::detail::r1::arena::process (tls=..., this=<optimized out>) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-df2a8868ba5d9111864e1a63085d7a4b/tbb-v2021.9.0/src/tbb/arena.cpp:137
#11 tbb::detail::r1::market::process (this=<optimized out>, j=...) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-df2a8868ba5d9111864e1a63085d7a4b/tbb-v2021.9.0/src/tbb/market.cpp:599
#12 0x00001507613d3cee in tbb::detail::r1::rml::private_worker::run (this=0x15075a0d5080) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-df2a8868ba5d9111864e1a63085d7a4b/tbb-v2021.9.0/src/tbb/private_server.cpp:271
#13 tbb::detail::r1::rml::private_worker::thread_routine (arg=0x15075a0d5080) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-df2a8868ba5d9111864e1a63085d7a4b/tbb-v2021.9.0/src/tbb/private_server.cpp:221
#14 0x000015075f6661ca in start_thread () from /lib64/libpthread.so.0
#15 0x000015075f2c18d3 in clone () from /lib64/libc.so.6
Thread 2 (Thread 0x1507358cd700 (LWP 1304112) "cmsRun"):
#0  0x000015075f6706a2 in waitpid () from /lib64/libpthread.so.0
#1  0x00001507599dfb67 in edm::service::cmssw_stacktrace_fork() () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02846/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-07-17-1200/lib/el8_amd64_gcc12/pluginFWCoreServicesPlugins.so
#2  0x00001507599e299a in edm::service::InitRootHandlers::stacktraceHelperThread() () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02846/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-07-17-1200/lib/el8_amd64_gcc12/pluginFWCoreServicesPlugins.so
#3  0x000015075fcd8a73 in std::execute_native_thread_routine (__p=0x150738c0cdb0) at ../../../../../libstdc++-v3/src/c++11/thread.cc:82
#4  0x000015075f6661ca in start_thread () from /lib64/libpthread.so.0
#5  0x000015075f2c18d3 in clone () from /lib64/libc.so.6
Thread 1 (Thread 0x1507612e1680 (LWP 1303317) "cmsRun"):
#0  0x000015075f3baac1 in poll () from /lib64/libc.so.6
#1  0x00001507599e2bb7 in edm::service::InitRootHandlers::stacktraceFromThread() () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02846/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-07-17-1200/lib/el8_amd64_gcc12/pluginFWCoreServicesPlugins.so
#2  0x00001507599e2db4 in sig_dostack_then_abort () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02846/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-07-17-1200/lib/el8_amd64_gcc12/pluginFWCoreServicesPlugins.so
#3  <signal handler called>
#4  tbb::detail::r1::spawn (t=..., ctx=...) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-df2a8868ba5d9111864e1a63085d7a4b/tbb-v2021.9.0/src/tbb/task_dispatcher.cpp:36
#5  0x0000150761f2d438 in tbb::detail::d1::function_task<edm::WaitingTaskHolder::doneWaiting(std::__exception_ptr::exception_ptr)::{lambda()#1}>::execute(tbb::detail::d1::execution_data&) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02846/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-07-17-1200/lib/el8_amd64_gcc12/libFWCoreFramework.so
#6  0x00001507613da3e1 in tbb::detail::r1::task_dispatcher::local_wait_for_all<false, tbb::detail::r1::external_waiter> (waiter=..., t=<optimized out>, this=0x15075ddbb900) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-df2a8868ba5d9111864e1a63085d7a4b/tbb-v2021.9.0/src/tbb/task_dispatcher.h:322
#7  tbb::detail::r1::task_dispatcher::local_wait_for_all<tbb::detail::r1::external_waiter> (waiter=..., t=<optimized out>, this=0x15075ddbb900) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-df2a8868ba5d9111864e1a63085d7a4b/tbb-v2021.9.0/src/tbb/task_dispatcher.h:458
#8  tbb::detail::r1::task_dispatcher::execute_and_wait (t=<optimized out>, wait_ctx=..., w_ctx=...) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-df2a8868ba5d9111864e1a63085d7a4b/tbb-v2021.9.0/src/tbb/task_dispatcher.cpp:168
#9  0x0000150761f433ab in edm::FinalWaitingTask::wait() () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02846/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-07-17-1200/lib/el8_amd64_gcc12/libFWCoreFramework.so
#10 0x0000150761f502f6 in edm::EventProcessor::endUnfinishedRun(bool) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02846/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-07-17-1200/lib/el8_amd64_gcc12/libFWCoreFramework.so
#11 0x0000150761f5164d in edm::EventProcessor::runToCompletion() () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02846/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-07-17-1200/lib/el8_amd64_gcc12/libFWCoreFramework.so
#12 0x0000000000408410 in tbb::detail::d1::task_arena_function<main::{lambda()#1}::operator()() const::{lambda()#1}, void>::operator()() const ()
#13 0x00001507613c69ad in tbb::detail::r1::task_arena_impl::execute (ta=..., d=...) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-df2a8868ba5d9111864e1a63085d7a4b/tbb-v2021.9.0/src/tbb/arena.cpp:688
#14 0x000000000040a102 in main::{lambda()#1}::operator()() const ()
#15 0x0000000000405100 in main ()

Current Modules:

Module: none (crashed)/data/cmsbld/jenkins/workspace/ib-run-qa/CMSSW_14_1_X_2024-07-17-2300/src/DQM/Integration/test/runtest.sh: line 28: 1303317 Segmentation fault      (core dumped) cmsRun $CLIENTS_DIR/$1 unitTest=True

---> test TestDQMOnlineClient-visualization_secondInstance had ERRORS

@iarspider
Copy link
Contributor

New occurance: CMSSW_14_1_X_2024-08-12-1100

A fatal system signal has occurred: segmentation violation
The following is the call stack containing the origin of the signal.

Mon Aug 12 18:43:19 CEST 2024
Thread 26 (Thread 0x148ea05ff700 (LWP 3757517) "cmsRun"):
#0  0x0000148f2766c48c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x0000148eee2dfba6 in Eigen::ThreadPoolTempl<tsl::thread::EigenEnvironment>::WaitForWork(Eigen::EventCount::Waiter*, tsl::thread::EigenEnvironment::Task*) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week0/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-08-12-1100/external/el8_amd64_gcc12/lib/libtensorflow_cc.so.2
#2  0x0000148eee2e01a5 in Eigen::ThreadPoolTempl<tsl::thread::EigenEnvironment>::WorkerLoop(int) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week0/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-08-12-1100/external/el8_amd64_gcc12/lib/libtensorflow_cc.so.2
#3  0x0000148eee2dd968 in std::_Function_handler<void (), tsl::thread::EigenEnvironment::CreateThread(std::function<void ()>)::{lambda()#1}>::_M_invoke(std::_Any_data const&) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week0/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-08-12-1100/external/el8_amd64_gcc12/lib/libtensorflow_cc.so.2
#4  0x0000148edd678472 in tsl::(anonymous namespace)::PThread::ThreadFn(void*) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week0/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-08-12-1100/external/el8_amd64_gcc12/lib/libtensorflow_framework.so.2
#5  0x0000148f276661ca in start_thread () from /lib64/libpthread.so.0
#6  0x0000148f272c18d3 in clone () from /lib64/libc.so.6
Thread 25 (Thread 0x148ea0fff700 (LWP 3757516) "cmsRun"):
#0  0x0000148f2766c48c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x0000148eee2dfba6 in Eigen::ThreadPoolTempl<tsl::thread::EigenEnvironment>::WaitForWork(Eigen::EventCount::Waiter*, tsl::thread::EigenEnvironment::Task*) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week0/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-08-12-1100/external/el8_amd64_gcc12/lib/libtensorflow_cc.so.2
#2  0x0000148eee2e01a5 in Eigen::ThreadPoolTempl<tsl::thread::EigenEnvironment>::WorkerLoop(int) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week0/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-08-12-1100/external/el8_amd64_gcc12/lib/libtensorflow_cc.so.2
#3  0x0000148eee2dd968 in std::_Function_handler<void (), tsl::thread::EigenEnvironment::CreateThread(std::function<void ()>)::{lambda()#1}>::_M_invoke(std::_Any_data const&) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week0/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-08-12-1100/external/el8_amd64_gcc12/lib/libtensorflow_cc.so.2
#4  0x0000148edd678472 in tsl::(anonymous namespace)::PThread::ThreadFn(void*) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week0/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-08-12-1100/external/el8_amd64_gcc12/lib/libtensorflow_framework.so.2
#5  0x0000148f276661ca in start_thread () from /lib64/libpthread.so.0
#6  0x0000148f272c18d3 in clone () from /lib64/libc.so.6
Thread 24 (Thread 0x148ea1525700 (LWP 3757515) "cmsRun"):
#0  0x0000148f2766c48c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x0000148eee2dfba6 in Eigen::ThreadPoolTempl<tsl::thread::EigenEnvironment>::WaitForWork(Eigen::EventCount::Waiter*, tsl::thread::EigenEnvironment::Task*) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week0/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-08-12-1100/external/el8_amd64_gcc12/lib/libtensorflow_cc.so.2
#2  0x0000148eee2e01a5 in Eigen::ThreadPoolTempl<tsl::thread::EigenEnvironment>::WorkerLoop(int) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week0/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-08-12-1100/external/el8_amd64_gcc12/lib/libtensorflow_cc.so.2
#3  0x0000148eee2dd968 in std::_Function_handler<void (), tsl::thread::EigenEnvironment::CreateThread(std::function<void ()>)::{lambda()#1}>::_M_invoke(std::_Any_data const&) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week0/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-08-12-1100/external/el8_amd64_gcc12/lib/libtensorflow_cc.so.2
#4  0x0000148edd678472 in tsl::(anonymous namespace)::PThread::ThreadFn(void*) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week0/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-08-12-1100/external/el8_amd64_gcc12/lib/libtensorflow_framework.so.2
#5  0x0000148f276661ca in start_thread () from /lib64/libpthread.so.0
#6  0x0000148f272c18d3 in clone () from /lib64/libc.so.6
Thread 23 (Thread 0x148ecd9f2700 (LWP 3757385) "cmsRun"):
#0  0x0000148f2766edd6 in do_futex_wait.constprop () from /lib64/libpthread.so.0
#1  0x0000148f2766eec8 in __new_sem_wait_slow.constprop.0 () from /lib64/libpthread.so.0
#2  0x0000148f1f5ce7c6 in XrdCl::JobManager::RunJobs() () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week0/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-08-12-1100/external/el8_amd64_gcc12/lib/libXrdCl.so.3
#3  0x0000148f1f5ce879 in RunRunnerThread () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week0/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-08-12-1100/external/el8_amd64_gcc12/lib/libXrdCl.so.3
#4  0x0000148f276661ca in start_thread () from /lib64/libpthread.so.0
#5  0x0000148f272c18d3 in clone () from /lib64/libc.so.6
Thread 22 (Thread 0x148ecdbf3700 (LWP 3757384) "cmsRun"):
#0  0x0000148f2766edd6 in do_futex_wait.constprop () from /lib64/libpthread.so.0
#1  0x0000148f2766eec8 in __new_sem_wait_slow.constprop.0 () from /lib64/libpthread.so.0
#2  0x0000148f1f5ce7c6 in XrdCl::JobManager::RunJobs() () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week0/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-08-12-1100/external/el8_amd64_gcc12/lib/libXrdCl.so.3
#3  0x0000148f1f5ce879 in RunRunnerThread () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week0/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-08-12-1100/external/el8_amd64_gcc12/lib/libXrdCl.so.3
#4  0x0000148f276661ca in start_thread () from /lib64/libpthread.so.0
#5  0x0000148f272c18d3 in clone () from /lib64/libc.so.6
Thread 21 (Thread 0x148ecddf4700 (LWP 3757383) "cmsRun"):
#0  0x0000148f2766edd6 in do_futex_wait.constprop () from /lib64/libpthread.so.0
#1  0x0000148f2766eec8 in __new_sem_wait_slow.constprop.0 () from /lib64/libpthread.so.0
#2  0x0000148f1f5ce7c6 in XrdCl::JobManager::RunJobs() () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week0/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-08-12-1100/external/el8_amd64_gcc12/lib/libXrdCl.so.3
#3  0x0000148f1f5ce879 in RunRunnerThread () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week0/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-08-12-1100/external/el8_amd64_gcc12/lib/libXrdCl.so.3
#4  0x0000148f276661ca in start_thread () from /lib64/libpthread.so.0
#5  0x0000148f272c18d3 in clone () from /lib64/libc.so.6
Thread 20 (Thread 0x148ecdff5700 (LWP 3757382) "cmsRun"):
#0  0x0000148f276701b0 in nanosleep () from /lib64/libpthread.so.0
#1  0x0000148f218e47e8 in XrdSysTimer::Wait(int) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week0/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-08-12-1100/external/el8_amd64_gcc12/lib/libXrdUtils.so.3
#2  0x0000148f1f53f37c in XrdCl::TaskManager::RunTasks() () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week0/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-08-12-1100/external/el8_amd64_gcc12/lib/libXrdCl.so.3
#3  0x0000148f1f53f4b9 in RunRunnerThread () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week0/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-08-12-1100/external/el8_amd64_gcc12/lib/libXrdCl.so.3
#4  0x0000148f276661ca in start_thread () from /lib64/libpthread.so.0
#5  0x0000148f272c18d3 in clone () from /lib64/libc.so.6
Thread 19 (Thread 0x148ece1f6700 (LWP 3757381) "cmsRun"):
#0  0x0000148f273c7307 in epoll_wait () from /lib64/libc.so.6
#1  0x0000148f218deb62 in XrdSys::IOEvents::PollE::Begin(XrdSysSemaphore*, int&, char const**) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week0/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-08-12-1100/external/el8_amd64_gcc12/lib/libXrdUtils.so.3
#2  0x0000148f218daacd in XrdSys::IOEvents::BootStrap::Start(void*) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week0/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-08-12-1100/external/el8_amd64_gcc12/lib/libXrdUtils.so.3
#3  0x0000148f218e3ef7 in XrdSysThread_Xeq () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week0/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-08-12-1100/external/el8_amd64_gcc12/lib/libXrdUtils.so.3
#4  0x0000148f276661ca in start_thread () from /lib64/libpthread.so.0
#5  0x0000148f272c18d3 in clone () from /lib64/libc.so.6
Thread 18 (Thread 0x148ece3f7700 (LWP 3757380) "cmsRun"):
#0  0x0000148f273c7307 in epoll_wait () from /lib64/libc.so.6
#1  0x0000148f218deb62 in XrdSys::IOEvents::PollE::Begin(XrdSysSemaphore*, int&, char const**) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week0/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-08-12-1100/external/el8_amd64_gcc12/lib/libXrdUtils.so.3
#2  0x0000148f218daacd in XrdSys::IOEvents::BootStrap::Start(void*) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week0/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-08-12-1100/external/el8_amd64_gcc12/lib/libXrdUtils.so.3
#3  0x0000148f218e3ef7 in XrdSysThread_Xeq () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week0/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-08-12-1100/external/el8_amd64_gcc12/lib/libXrdUtils.so.3
#4  0x0000148f276661ca in start_thread () from /lib64/libpthread.so.0
#5  0x0000148f272c18d3 in clone () from /lib64/libc.so.6
Thread 17 (Thread 0x148ece5f8700 (LWP 3757379) "cmsRun"):
#0  0x0000148f273c7307 in epoll_wait () from /lib64/libc.so.6
#1  0x0000148f218deb62 in XrdSys::IOEvents::PollE::Begin(XrdSysSemaphore*, int&, char const**) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week0/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-08-12-1100/external/el8_amd64_gcc12/lib/libXrdUtils.so.3
#2  0x0000148f218daacd in XrdSys::IOEvents::BootStrap::Start(void*) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week0/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-08-12-1100/external/el8_amd64_gcc12/lib/libXrdUtils.so.3
#3  0x0000148f218e3ef7 in XrdSysThread_Xeq () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week0/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-08-12-1100/external/el8_amd64_gcc12/lib/libXrdUtils.so.3
#4  0x0000148f276661ca in start_thread () from /lib64/libpthread.so.0
#5  0x0000148f272c18d3 in clone () from /lib64/libc.so.6
Thread 16 (Thread 0x148ece7f9700 (LWP 3757378) "cmsRun"):
#0  0x0000148f273c7307 in epoll_wait () from /lib64/libc.so.6
#1  0x0000148f218deb62 in XrdSys::IOEvents::PollE::Begin(XrdSysSemaphore*, int&, char const**) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week0/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-08-12-1100/external/el8_amd64_gcc12/lib/libXrdUtils.so.3
#2  0x0000148f218daacd in XrdSys::IOEvents::BootStrap::Start(void*) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week0/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-08-12-1100/external/el8_amd64_gcc12/lib/libXrdUtils.so.3
#3  0x0000148f218e3ef7 in XrdSysThread_Xeq () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week0/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-08-12-1100/external/el8_amd64_gcc12/lib/libXrdUtils.so.3
#4  0x0000148f276661ca in start_thread () from /lib64/libpthread.so.0
#5  0x0000148f272c18d3 in clone () from /lib64/libc.so.6
Thread 15 (Thread 0x148ece9fa700 (LWP 3757377) "cmsRun"):
#0  0x0000148f273c7307 in epoll_wait () from /lib64/libc.so.6
#1  0x0000148f218deb62 in XrdSys::IOEvents::PollE::Begin(XrdSysSemaphore*, int&, char const**) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week0/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-08-12-1100/external/el8_amd64_gcc12/lib/libXrdUtils.so.3
#2  0x0000148f218daacd in XrdSys::IOEvents::BootStrap::Start(void*) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week0/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-08-12-1100/external/el8_amd64_gcc12/lib/libXrdUtils.so.3
#3  0x0000148f218e3ef7 in XrdSysThread_Xeq () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week0/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-08-12-1100/external/el8_amd64_gcc12/lib/libXrdUtils.so.3
#4  0x0000148f276661ca in start_thread () from /lib64/libpthread.so.0
#5  0x0000148f272c18d3 in clone () from /lib64/libc.so.6
Thread 14 (Thread 0x148ecebfb700 (LWP 3757376) "cmsRun"):
#0  0x0000148f273c7307 in epoll_wait () from /lib64/libc.so.6
#1  0x0000148f218deb62 in XrdSys::IOEvents::PollE::Begin(XrdSysSemaphore*, int&, char const**) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week0/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-08-12-1100/external/el8_amd64_gcc12/lib/libXrdUtils.so.3
#2  0x0000148f218daacd in XrdSys::IOEvents::BootStrap::Start(void*) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week0/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-08-12-1100/external/el8_amd64_gcc12/lib/libXrdUtils.so.3
#3  0x0000148f218e3ef7 in XrdSysThread_Xeq () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week0/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-08-12-1100/external/el8_amd64_gcc12/lib/libXrdUtils.so.3
#4  0x0000148f276661ca in start_thread () from /lib64/libpthread.so.0
#5  0x0000148f272c18d3 in clone () from /lib64/libc.so.6
Thread 13 (Thread 0x148ecedfc700 (LWP 3757375) "cmsRun"):
#0  0x0000148f273c7307 in epoll_wait () from /lib64/libc.so.6
#1  0x0000148f218deb62 in XrdSys::IOEvents::PollE::Begin(XrdSysSemaphore*, int&, char const**) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week0/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-08-12-1100/external/el8_amd64_gcc12/lib/libXrdUtils.so.3
#2  0x0000148f218daacd in XrdSys::IOEvents::BootStrap::Start(void*) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week0/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-08-12-1100/external/el8_amd64_gcc12/lib/libXrdUtils.so.3
#3  0x0000148f218e3ef7 in XrdSysThread_Xeq () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week0/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-08-12-1100/external/el8_amd64_gcc12/lib/libXrdUtils.so.3
#4  0x0000148f276661ca in start_thread () from /lib64/libpthread.so.0
#5  0x0000148f272c18d3 in clone () from /lib64/libc.so.6
Thread 12 (Thread 0x148eceffd700 (LWP 3757374) "cmsRun"):
#0  0x0000148f273c7307 in epoll_wait () from /lib64/libc.so.6
#1  0x0000148f218deb62 in XrdSys::IOEvents::PollE::Begin(XrdSysSemaphore*, int&, char const**) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week0/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-08-12-1100/external/el8_amd64_gcc12/lib/libXrdUtils.so.3
#2  0x0000148f218daacd in XrdSys::IOEvents::BootStrap::Start(void*) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week0/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-08-12-1100/external/el8_amd64_gcc12/lib/libXrdUtils.so.3
#3  0x0000148f218e3ef7 in XrdSysThread_Xeq () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week0/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-08-12-1100/external/el8_amd64_gcc12/lib/libXrdUtils.so.3
#4  0x0000148f276661ca in start_thread () from /lib64/libpthread.so.0
#5  0x0000148f272c18d3 in clone () from /lib64/libc.so.6
Thread 11 (Thread 0x148ecf1fe700 (LWP 3757373) "cmsRun"):
#0  0x0000148f273c7307 in epoll_wait () from /lib64/libc.so.6
#1  0x0000148f218deb62 in XrdSys::IOEvents::PollE::Begin(XrdSysSemaphore*, int&, char const**) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week0/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-08-12-1100/external/el8_amd64_gcc12/lib/libXrdUtils.so.3
#2  0x0000148f218daacd in XrdSys::IOEvents::BootStrap::Start(void*) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week0/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-08-12-1100/external/el8_amd64_gcc12/lib/libXrdUtils.so.3
#3  0x0000148f218e3ef7 in XrdSysThread_Xeq () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week0/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-08-12-1100/external/el8_amd64_gcc12/lib/libXrdUtils.so.3
#4  0x0000148f276661ca in start_thread () from /lib64/libpthread.so.0
#5  0x0000148f272c18d3 in clone () from /lib64/libc.so.6
Thread 10 (Thread 0x148ecf3ff700 (LWP 3757372) "cmsRun"):
#0  0x0000148f273c7307 in epoll_wait () from /lib64/libc.so.6
#1  0x0000148f218deb62 in XrdSys::IOEvents::PollE::Begin(XrdSysSemaphore*, int&, char const**) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week0/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-08-12-1100/external/el8_amd64_gcc12/lib/libXrdUtils.so.3
#2  0x0000148f218daacd in XrdSys::IOEvents::BootStrap::Start(void*) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week0/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-08-12-1100/external/el8_amd64_gcc12/lib/libXrdUtils.so.3
#3  0x0000148f218e3ef7 in XrdSysThread_Xeq () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week0/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-08-12-1100/external/el8_amd64_gcc12/lib/libXrdUtils.so.3
#4  0x0000148f276661ca in start_thread () from /lib64/libpthread.so.0
#5  0x0000148f272c18d3 in clone () from /lib64/libc.so.6
Thread 9 (Thread 0x148ed03ff700 (LWP 3757371) "cmsRun"):
#0  0x0000148f27390098 in nanosleep () from /lib64/libc.so.6
#1  0x0000148f2738ff9e in sleep () from /lib64/libc.so.6
#2  0x0000148f21a3c1d0 in sig_pause_for_stacktrace () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week0/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-08-12-1100/lib/el8_amd64_gcc12/pluginFWCoreServicesPlugins.so
#3  <signal handler called>
#4  0x0000148f272c141d in syscall () from /lib64/libc.so.6
#5  0x0000148f2936b46b in tbb::detail::r1::futex_wakeup_one (futex=0x148f22030024) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-2391c941213c757dc9a1835b31681235/tbb-v2021.9.0/src/tbb/semaphore.h:109
#6  tbb::detail::r1::binary_semaphore::V (this=0x148f22030024) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-2391c941213c757dc9a1835b31681235/tbb-v2021.9.0/src/tbb/semaphore.h:262
#7  tbb::detail::r1::rml::internal::thread_monitor::notify (this=0x148f22030020) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-2391c941213c757dc9a1835b31681235/tbb-v2021.9.0/src/tbb/rml_thread_monitor.h:230
#8  tbb::detail::r1::rml::private_worker::wake_or_launch (this=0x148f22030000) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-2391c941213c757dc9a1835b31681235/tbb-v2021.9.0/src/tbb/private_server.cpp:292
#9  tbb::detail::r1::rml::private_server::wake_some (this=<optimized out>, additional_slack=<optimized out>) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-2391c941213c757dc9a1835b31681235/tbb-v2021.9.0/src/tbb/private_server.cpp:412
#10 0x0000148f2936be49 in tbb::detail::r1::rml::private_server::propagate_chain_reaction (this=<optimized out>) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-2391c941213c757dc9a1835b31681235/tbb-v2021.9.0/src/tbb/private_server.cpp:159
#11 tbb::detail::r1::rml::private_worker::run (this=0x148f2202fe00) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-2391c941213c757dc9a1835b31681235/tbb-v2021.9.0/src/tbb/private_server.cpp:275
#12 tbb::detail::r1::rml::private_worker::thread_routine (arg=0x148f2202fe00) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-2391c941213c757dc9a1835b31681235/tbb-v2021.9.0/src/tbb/private_server.cpp:221
#13 0x0000148f276661ca in start_thread () from /lib64/libpthread.so.0
#14 0x0000148f272c18d3 in clone () from /lib64/libc.so.6
Thread 8 (Thread 0x148ed13fd700 (LWP 3757370) "cmsRun"):
#0  0x0000148f27390098 in nanosleep () from /lib64/libc.so.6
#1  0x0000148f2738ff9e in sleep () from /lib64/libc.so.6
#2  0x0000148f21a3c1d0 in sig_pause_for_stacktrace () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week0/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-08-12-1100/lib/el8_amd64_gcc12/pluginFWCoreServicesPlugins.so
#3  <signal handler called>
#4  0x0000148f29f5f956 in edm::StreamSchedule::processOneStreamAsync<edm::OccurrenceTraits<edm::RunPrincipal, (edm::BranchActionType)2> >(edm::WaitingTaskHolder, edm::OccurrenceTraits<edm::RunPrincipal, (edm::BranchActionType)2>::TransitionInfoType&, edm::ServiceToken const&, bool)::{lambda()#2}::operator()() () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week0/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-08-12-1100/lib/el8_amd64_gcc12/libFWCoreFramework.so
#5  0x0000148f29f6ee37 in tbb::detail::d1::enqueue_task<edm::StreamSchedule::processOneStreamAsync<edm::OccurrenceTraits<edm::RunPrincipal, (edm::BranchActionType)2> >(edm::WaitingTaskHolder, edm::OccurrenceTraits<edm::RunPrincipal, (edm::BranchActionType)2>::TransitionInfoType&, edm::ServiceToken const&, bool)::{lambda()#4}>::execute(tbb::detail::d1::execution_data&) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week0/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-08-12-1100/lib/el8_amd64_gcc12/libFWCoreFramework.so
#6  0x0000148f29369b3b in tbb::detail::r1::task_dispatcher::local_wait_for_all<false, tbb::detail::r1::outermost_worker_waiter> (t=0x148eb6b88c00, waiter=..., this=0x148f25dbbb80) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-2391c941213c757dc9a1835b31681235/tbb-v2021.9.0/src/tbb/task_dispatcher.h:322
#7  tbb::detail::r1::task_dispatcher::local_wait_for_all<tbb::detail::r1::outermost_worker_waiter> (t=0x0, waiter=..., this=0x148f25dbbb80) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-2391c941213c757dc9a1835b31681235/tbb-v2021.9.0/src/tbb/task_dispatcher.h:458
#8  tbb::detail::r1::arena::process (tls=..., this=<optimized out>) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-2391c941213c757dc9a1835b31681235/tbb-v2021.9.0/src/tbb/arena.cpp:137
#9  tbb::detail::r1::market::process (this=<optimized out>, j=...) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-2391c941213c757dc9a1835b31681235/tbb-v2021.9.0/src/tbb/market.cpp:599
#10 0x0000148f2936bcee in tbb::detail::r1::rml::private_worker::run (this=0x148f2202ff00) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-2391c941213c757dc9a1835b31681235/tbb-v2021.9.0/src/tbb/private_server.cpp:271
#11 tbb::detail::r1::rml::private_worker::thread_routine (arg=0x148f2202ff00) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-2391c941213c757dc9a1835b31681235/tbb-v2021.9.0/src/tbb/private_server.cpp:221
#12 0x0000148f276661ca in start_thread () from /lib64/libpthread.so.0
#13 0x0000148f272c18d3 in clone () from /lib64/libc.so.6
Thread 7 (Thread 0x148ed1dfe700 (LWP 3757369) "cmsRun"):
#0  0x0000148f273baac1 in poll () from /lib64/libc.so.6
#1  0x0000148f21a3f857 in edm::service::InitRootHandlers::stacktraceFromThread() () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week0/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-08-12-1100/lib/el8_amd64_gcc12/pluginFWCoreServicesPlugins.so
#2  0x0000148f21a3fa54 in sig_dostack_then_abort () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week0/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-08-12-1100/lib/el8_amd64_gcc12/pluginFWCoreServicesPlugins.so
#3  <signal handler called>
#4  tbb::detail::r1::spawn (t=..., ctx=...) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-2391c941213c757dc9a1835b31681235/tbb-v2021.9.0/src/tbb/task_dispatcher.cpp:36
#5  0x0000148f29f2e5e8 in tbb::detail::d1::function_task<edm::WaitingTaskHolder::doneWaiting(std::__exception_ptr::exception_ptr)::{lambda()#1}>::execute(tbb::detail::d1::execution_data&) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week0/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-08-12-1100/lib/el8_amd64_gcc12/libFWCoreFramework.so
#6  0x0000148f29369b3b in tbb::detail::r1::task_dispatcher::local_wait_for_all<false, tbb::detail::r1::outermost_worker_waiter> (t=0x148eb6b88900, waiter=..., this=0x148f25dbba80) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-2391c941213c757dc9a1835b31681235/tbb-v2021.9.0/src/tbb/task_dispatcher.h:322
#7  tbb::detail::r1::task_dispatcher::local_wait_for_all<tbb::detail::r1::outermost_worker_waiter> (t=0x0, waiter=..., this=0x148f25dbba80) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-2391c941213c757dc9a1835b31681235/tbb-v2021.9.0/src/tbb/task_dispatcher.h:458
#8  tbb::detail::r1::arena::process (tls=..., this=<optimized out>) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-2391c941213c757dc9a1835b31681235/tbb-v2021.9.0/src/tbb/arena.cpp:137
#9  tbb::detail::r1::market::process (this=<optimized out>, j=...) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-2391c941213c757dc9a1835b31681235/tbb-v2021.9.0/src/tbb/market.cpp:599
#10 0x0000148f2936bcee in tbb::detail::r1::rml::private_worker::run (this=0x148f22030000) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-2391c941213c757dc9a1835b31681235/tbb-v2021.9.0/src/tbb/private_server.cpp:271
#11 tbb::detail::r1::rml::private_worker::thread_routine (arg=0x148f22030000) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-2391c941213c757dc9a1835b31681235/tbb-v2021.9.0/src/tbb/private_server.cpp:221
#12 0x0000148f276661ca in start_thread () from /lib64/libpthread.so.0
#13 0x0000148f272c18d3 in clone () from /lib64/libc.so.6
Thread 6 (Thread 0x148ed27ff700 (LWP 3757368) "cmsRun"):
#0  0x0000148f27390098 in nanosleep () from /lib64/libc.so.6
#1  0x0000148f2738ff9e in sleep () from /lib64/libc.so.6
#2  0x0000148f21a3c1d0 in sig_pause_for_stacktrace () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week0/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-08-12-1100/lib/el8_amd64_gcc12/pluginFWCoreServicesPlugins.so
#3  <signal handler called>
#4  0x0000148f29f5f33f in edm::StreamSchedule::processOneStreamAsync<edm::OccurrenceTraits<edm::RunPrincipal, (edm::BranchActionType)2> >(edm::WaitingTaskHolder, edm::OccurrenceTraits<edm::RunPrincipal, (edm::BranchActionType)2>::TransitionInfoType&, edm::ServiceToken const&, bool)::{lambda()#2}::operator()() () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week0/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-08-12-1100/lib/el8_amd64_gcc12/libFWCoreFramework.so
#5  0x0000148f29f6ee37 in tbb::detail::d1::enqueue_task<edm::StreamSchedule::processOneStreamAsync<edm::OccurrenceTraits<edm::RunPrincipal, (edm::BranchActionType)2> >(edm::WaitingTaskHolder, edm::OccurrenceTraits<edm::RunPrincipal, (edm::BranchActionType)2>::TransitionInfoType&, edm::ServiceToken const&, bool)::{lambda()#4}>::execute(tbb::detail::d1::execution_data&) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week0/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-08-12-1100/lib/el8_amd64_gcc12/libFWCoreFramework.so
#6  0x0000148f29369b3b in tbb::detail::r1::task_dispatcher::local_wait_for_all<false, tbb::detail::r1::outermost_worker_waiter> (t=0x148eb6b89100, waiter=..., this=0x148f25dbb980) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-2391c941213c757dc9a1835b31681235/tbb-v2021.9.0/src/tbb/task_dispatcher.h:322
#7  tbb::detail::r1::task_dispatcher::local_wait_for_all<tbb::detail::r1::outermost_worker_waiter> (t=0x0, waiter=..., this=0x148f25dbb980) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-2391c941213c757dc9a1835b31681235/tbb-v2021.9.0/src/tbb/task_dispatcher.h:458
#8  tbb::detail::r1::arena::process (tls=..., this=<optimized out>) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-2391c941213c757dc9a1835b31681235/tbb-v2021.9.0/src/tbb/arena.cpp:137
#9  tbb::detail::r1::market::process (this=<optimized out>, j=...) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-2391c941213c757dc9a1835b31681235/tbb-v2021.9.0/src/tbb/market.cpp:599
#10 0x0000148f2936bcee in tbb::detail::r1::rml::private_worker::run (this=0x148f2202fe80) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-2391c941213c757dc9a1835b31681235/tbb-v2021.9.0/src/tbb/private_server.cpp:271
#11 tbb::detail::r1::rml::private_worker::thread_routine (arg=0x148f2202fe80) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-2391c941213c757dc9a1835b31681235/tbb-v2021.9.0/src/tbb/private_server.cpp:221
#12 0x0000148f276661ca in start_thread () from /lib64/libpthread.so.0
#13 0x0000148f272c18d3 in clone () from /lib64/libc.so.6
Thread 5 (Thread 0x148ed37ff700 (LWP 3757367) "cmsRun"):
#0  0x0000148f27390098 in nanosleep () from /lib64/libc.so.6
#1  0x0000148f2738ff9e in sleep () from /lib64/libc.so.6
#2  0x0000148f21a3c1d0 in sig_pause_for_stacktrace () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week0/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-08-12-1100/lib/el8_amd64_gcc12/pluginFWCoreServicesPlugins.so
#3  <signal handler called>
#4  0x0000148f29f5f33f in edm::StreamSchedule::processOneStreamAsync<edm::OccurrenceTraits<edm::RunPrincipal, (edm::BranchActionType)2> >(edm::WaitingTaskHolder, edm::OccurrenceTraits<edm::RunPrincipal, (edm::BranchActionType)2>::TransitionInfoType&, edm::ServiceToken const&, bool)::{lambda()#2}::operator()() () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week0/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-08-12-1100/lib/el8_amd64_gcc12/libFWCoreFramework.so
#5  0x0000148f29f6ee37 in tbb::detail::d1::enqueue_task<edm::StreamSchedule::processOneStreamAsync<edm::OccurrenceTraits<edm::RunPrincipal, (edm::BranchActionType)2> >(edm::WaitingTaskHolder, edm::OccurrenceTraits<edm::RunPrincipal, (edm::BranchActionType)2>::TransitionInfoType&, edm::ServiceToken const&, bool)::{lambda()#4}>::execute(tbb::detail::d1::execution_data&) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week0/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-08-12-1100/lib/el8_amd64_gcc12/libFWCoreFramework.so
#6  0x0000148f29369b3b in tbb::detail::r1::task_dispatcher::local_wait_for_all<false, tbb::detail::r1::outermost_worker_waiter> (t=0x148eb6b88300, waiter=..., this=0x148f25dbbc00) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-2391c941213c757dc9a1835b31681235/tbb-v2021.9.0/src/tbb/task_dispatcher.h:322
#7  tbb::detail::r1::task_dispatcher::local_wait_for_all<tbb::detail::r1::outermost_worker_waiter> (t=0x0, waiter=..., this=0x148f25dbbc00) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-2391c941213c757dc9a1835b31681235/tbb-v2021.9.0/src/tbb/task_dispatcher.h:458
#8  tbb::detail::r1::arena::process (tls=..., this=<optimized out>) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-2391c941213c757dc9a1835b31681235/tbb-v2021.9.0/src/tbb/arena.cpp:137
#9  tbb::detail::r1::market::process (this=<optimized out>, j=...) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-2391c941213c757dc9a1835b31681235/tbb-v2021.9.0/src/tbb/market.cpp:599
#10 0x0000148f2936bcee in tbb::detail::r1::rml::private_worker::run (this=0x148f2202ff80) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-2391c941213c757dc9a1835b31681235/tbb-v2021.9.0/src/tbb/private_server.cpp:271
#11 tbb::detail::r1::rml::private_worker::thread_routine (arg=0x148f2202ff80) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-2391c941213c757dc9a1835b31681235/tbb-v2021.9.0/src/tbb/private_server.cpp:221
#12 0x0000148f276661ca in start_thread () from /lib64/libpthread.so.0
#13 0x0000148f272c18d3 in clone () from /lib64/libc.so.6
Thread 4 (Thread 0x148ed474c700 (LWP 3757366) "cmsRun"):
#0  0x0000148f27390098 in nanosleep () from /lib64/libc.so.6
#1  0x0000148f2738ff9e in sleep () from /lib64/libc.so.6
#2  0x0000148f21a3c1d0 in sig_pause_for_stacktrace () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week0/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-08-12-1100/lib/el8_amd64_gcc12/pluginFWCoreServicesPlugins.so
#3  <signal handler called>
#4  0x0000148f29f5cf4d in std::__weak_ptr<edm::serviceregistry::ServicesManager, (__gnu_cxx::_Lock_policy)2>::__weak_ptr(std::__weak_ptr<edm::serviceregistry::ServicesManager, (__gnu_cxx::_Lock_policy)2> const&) [clone .lto_priv.0] () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week0/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-08-12-1100/lib/el8_amd64_gcc12/libFWCoreFramework.so
#5  0x0000148f29f5f9f6 in edm::StreamSchedule::processOneStreamAsync<edm::OccurrenceTraits<edm::RunPrincipal, (edm::BranchActionType)2> >(edm::WaitingTaskHolder, edm::OccurrenceTraits<edm::RunPrincipal, (edm::BranchActionType)2>::TransitionInfoType&, edm::ServiceToken const&, bool)::{lambda()#2}::operator()() () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week0/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-08-12-1100/lib/el8_amd64_gcc12/libFWCoreFramework.so
#6  0x0000148f29f6eead in tbb::detail::d1::function_task<edm::StreamSchedule::processOneStreamAsync<edm::OccurrenceTraits<edm::RunPrincipal, (edm::BranchActionType)2> >(edm::WaitingTaskHolder, edm::OccurrenceTraits<edm::RunPrincipal, (edm::BranchActionType)2>::TransitionInfoType&, edm::ServiceToken const&, bool)::{lambda()#3}>::execute(tbb::detail::d1::execution_data&) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week0/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-08-12-1100/lib/el8_amd64_gcc12/libFWCoreFramework.so
#7  0x0000148f29369b3b in tbb::detail::r1::task_dispatcher::local_wait_for_all<false, tbb::detail::r1::outermost_worker_waiter> (t=0x148e8c7d4600, waiter=..., this=0x148f25dbbb00) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-2391c941213c757dc9a1835b31681235/tbb-v2021.9.0/src/tbb/task_dispatcher.h:322
#8  tbb::detail::r1::task_dispatcher::local_wait_for_all<tbb::detail::r1::outermost_worker_waiter> (t=0x0, waiter=..., this=0x148f25dbbb00) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-2391c941213c757dc9a1835b31681235/tbb-v2021.9.0/src/tbb/task_dispatcher.h:458
#9  tbb::detail::r1::arena::process (tls=..., this=<optimized out>) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-2391c941213c757dc9a1835b31681235/tbb-v2021.9.0/src/tbb/arena.cpp:137
#10 tbb::detail::r1::market::process (this=<optimized out>, j=...) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-2391c941213c757dc9a1835b31681235/tbb-v2021.9.0/src/tbb/market.cpp:599
#11 0x0000148f2936bcee in tbb::detail::r1::rml::private_worker::run (this=0x148f22030100) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-2391c941213c757dc9a1835b31681235/tbb-v2021.9.0/src/tbb/private_server.cpp:271
#12 tbb::detail::r1::rml::private_worker::thread_routine (arg=0x148f22030100) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-2391c941213c757dc9a1835b31681235/tbb-v2021.9.0/src/tbb/private_server.cpp:221
#13 0x0000148f276661ca in start_thread () from /lib64/libpthread.so.0
#14 0x0000148f272c18d3 in clone () from /lib64/libc.so.6
Thread 3 (Thread 0x148ed514d700 (LWP 3757365) "cmsRun"):
#0  0x0000148f27390098 in nanosleep () from /lib64/libc.so.6
#1  0x0000148f2738ff9e in sleep () from /lib64/libc.so.6
#2  0x0000148f21a3c1d0 in sig_pause_for_stacktrace () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week0/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-08-12-1100/lib/el8_amd64_gcc12/pluginFWCoreServicesPlugins.so
#3  <signal handler called>
#4  0x0000148f272c0b8b in sched_yield () from /lib64/libc.so.6
#5  0x0000148f2935dc73 in __gthread_yield () at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/el8_amd64_gcc12/external/gcc/12.3.1-40d504be6370b5a30e3947a6e575ca28/include/c++/12.3.1/x86_64-redhat-linux-gnu/bits/gthr-default.h:693
#6  std::this_thread::yield () at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/el8_amd64_gcc12/external/gcc/12.3.1-40d504be6370b5a30e3947a6e575ca28/include/c++/12.3.1/bits/std_thread.h:353
#7  tbb::detail::r1::stealing_loop_backoff::pause (this=0x148ed5146ef8) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-2391c941213c757dc9a1835b31681235/tbb-v2021.9.0/src/tbb/scheduler_common.h:266
#8  tbb::detail::r1::waiter_base::pause (this=0x148ed5146ef0) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-2391c941213c757dc9a1835b31681235/tbb-v2021.9.0/src/tbb/waiters.h:35
#9  tbb::detail::r1::outermost_worker_waiter::pause (this=0x148ed5146ef0) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-2391c941213c757dc9a1835b31681235/tbb-v2021.9.0/src/tbb/waiters.h:69
#10 tbb::detail::r1::task_dispatcher::receive_or_steal_task<true, tbb::detail::r1::outermost_worker_waiter> (this=<optimized out>, tls=..., ed=..., waiter=..., isolation=<optimized out>, fifo_allowed=<optimized out>, critical_allowed=<optimized out>) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-2391c941213c757dc9a1835b31681235/tbb-v2021.9.0/src/tbb/task_dispatcher.h:231
#11 0x0000148f29369cb2 in tbb::detail::r1::task_dispatcher::local_wait_for_all<false, tbb::detail::r1::outermost_worker_waiter> (t=0x0, waiter=..., this=0x148f25dbba00) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-2391c941213c757dc9a1835b31681235/tbb-v2021.9.0/src/tbb/task_dispatcher.h:350
#12 tbb::detail::r1::task_dispatcher::local_wait_for_all<tbb::detail::r1::outermost_worker_waiter> (t=0x0, waiter=..., this=0x148f25dbba00) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-2391c941213c757dc9a1835b31681235/tbb-v2021.9.0/src/tbb/task_dispatcher.h:458
#13 tbb::detail::r1::arena::process (tls=..., this=<optimized out>) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-2391c941213c757dc9a1835b31681235/tbb-v2021.9.0/src/tbb/arena.cpp:137
#14 tbb::detail::r1::market::process (this=<optimized out>, j=...) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-2391c941213c757dc9a1835b31681235/tbb-v2021.9.0/src/tbb/market.cpp:599
#15 0x0000148f2936bcee in tbb::detail::r1::rml::private_worker::run (this=0x148f22030080) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-2391c941213c757dc9a1835b31681235/tbb-v2021.9.0/src/tbb/private_server.cpp:271
#16 tbb::detail::r1::rml::private_worker::thread_routine (arg=0x148f22030080) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-2391c941213c757dc9a1835b31681235/tbb-v2021.9.0/src/tbb/private_server.cpp:221
#17 0x0000148f276661ca in start_thread () from /lib64/libpthread.so.0
#18 0x0000148f272c18d3 in clone () from /lib64/libc.so.6
Thread 2 (Thread 0x148efd5ff700 (LWP 3757295) "cmsRun"):
#0  0x0000148f276706a2 in waitpid () from /lib64/libpthread.so.0
#1  0x0000148f21a3c327 in edm::service::cmssw_stacktrace_fork() () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week0/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-08-12-1100/lib/el8_amd64_gcc12/pluginFWCoreServicesPlugins.so
#2  0x0000148f21a3f63a in edm::service::InitRootHandlers::stacktraceHelperThread() () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week0/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-08-12-1100/lib/el8_amd64_gcc12/pluginFWCoreServicesPlugins.so
#3  0x0000148f27cd8a73 in std::execute_native_thread_routine (__p=0x148f00f2be70) at ../../../../../libstdc++-v3/src/c++11/thread.cc:82
#4  0x0000148f276661ca in start_thread () from /lib64/libpthread.so.0
#5  0x0000148f272c18d3 in clone () from /lib64/libc.so.6
Thread 1 (Thread 0x148f289fe680 (LWP 3756508) "cmsRun"):
#0  0x0000148f27390098 in nanosleep () from /lib64/libc.so.6
#1  0x0000148f2738ff9e in sleep () from /lib64/libc.so.6
#2  0x0000148f21a3c1d0 in sig_pause_for_stacktrace () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week0/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-08-12-1100/lib/el8_amd64_gcc12/pluginFWCoreServicesPlugins.so
#3  <signal handler called>
#4  0x0000148f272c0b8b in sched_yield () from /lib64/libc.so.6
#5  0x0000148f29370686 in __gthread_yield () at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/el8_amd64_gcc12/external/gcc/12.3.1-40d504be6370b5a30e3947a6e575ca28/include/c++/12.3.1/x86_64-redhat-linux-gnu/bits/gthr-default.h:693
#6  std::this_thread::yield () at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/el8_amd64_gcc12/external/gcc/12.3.1-40d504be6370b5a30e3947a6e575ca28/include/c++/12.3.1/bits/std_thread.h:353
#7  tbb::detail::r1::stealing_loop_backoff::pause (this=0x7ffeeb9dc0e8) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-2391c941213c757dc9a1835b31681235/tbb-v2021.9.0/src/tbb/scheduler_common.h:266
#8  tbb::detail::r1::waiter_base::pause (this=0x7ffeeb9dc0e0) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-2391c941213c757dc9a1835b31681235/tbb-v2021.9.0/src/tbb/waiters.h:35
#9  tbb::detail::r1::external_waiter::pause (this=0x7ffeeb9dc0e0) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-2391c941213c757dc9a1835b31681235/tbb-v2021.9.0/src/tbb/waiters.h:138
#10 tbb::detail::r1::task_dispatcher::receive_or_steal_task<true, tbb::detail::r1::external_waiter> (this=<optimized out>, tls=..., ed=..., waiter=..., isolation=<optimized out>, fifo_allowed=<optimized out>, critical_allowed=<optimized out>) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-2391c941213c757dc9a1835b31681235/tbb-v2021.9.0/src/tbb/task_dispatcher.h:231
#11 0x0000148f293724e2 in tbb::detail::r1::task_dispatcher::local_wait_for_all<false, tbb::detail::r1::external_waiter> (waiter=..., t=0x0, this=0x148f25dbb900) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-2391c941213c757dc9a1835b31681235/tbb-v2021.9.0/src/tbb/task_dispatcher.h:350
#12 tbb::detail::r1::task_dispatcher::local_wait_for_all<tbb::detail::r1::external_waiter> (waiter=..., t=<optimized out>, this=0x148f25dbb900) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-2391c941213c757dc9a1835b31681235/tbb-v2021.9.0/src/tbb/task_dispatcher.h:458
#13 tbb::detail::r1::task_dispatcher::execute_and_wait (t=<optimized out>, wait_ctx=..., w_ctx=...) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-2391c941213c757dc9a1835b31681235/tbb-v2021.9.0/src/tbb/task_dispatcher.cpp:168
#14 0x0000148f29f445fb in edm::FinalWaitingTask::wait() () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week0/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-08-12-1100/lib/el8_amd64_gcc12/libFWCoreFramework.so
#15 0x0000148f29f51296 in edm::EventProcessor::endUnfinishedRun(bool) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week0/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-08-12-1100/lib/el8_amd64_gcc12/libFWCoreFramework.so
#16 0x0000148f29f525ed in edm::EventProcessor::runToCompletion() () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week0/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-08-12-1100/lib/el8_amd64_gcc12/libFWCoreFramework.so
#17 0x000000000040840c in tbb::detail::d1::task_arena_function<main::{lambda()#1}::operator()() const::{lambda()#1}, void>::operator()() const ()
#18 0x0000148f2935e9ad in tbb::detail::r1::task_arena_impl::execute (ta=..., d=...) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-2391c941213c757dc9a1835b31681235/tbb-v2021.9.0/src/tbb/arena.cpp:688
#19 0x000000000040a0f2 in main::{lambda()#1}::operator()() const ()
#20 0x0000000000405100 in main ()

Current Modules:

Module: none (crashed)
Module: none
Module: none
Module: none
Module: none
Module: none
Module: none
Module: none

@iarspider
Copy link
Contributor

iarspider commented Aug 19, 2024

Another occurance: el8_amd64_gcc12/CMSSW_14_1_MULTIARCHS_X_2024-08-18-0000

A fatal system signal has occurred: segmentation violation
The following is the call stack containing the origin of the signal.

Sun Aug 18 13:15:09 CEST 2024
Thread 26 (Thread 0x1539aa5ff700 (LWP 3435629) "cmsRun"):
#0  0x0000153a3206c48c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00001539f688d9a6 in Eigen::ThreadPoolTempl<tsl::thread::EigenEnvironment>::WaitForWork(Eigen::EventCount::Waiter*, tsl::thread::EigenEnvironment::Task*) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week1/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_MULTIARCHS_X_2024-08-18-0000/external/el8_amd64_gcc12/lib/scram_x86-64-v3/libtensorflow_cc.so.2
#2  0x00001539f688dfa5 in Eigen::ThreadPoolTempl<tsl::thread::EigenEnvironment>::WorkerLoop(int) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week1/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_MULTIARCHS_X_2024-08-18-0000/external/el8_amd64_gcc12/lib/scram_x86-64-v3/libtensorflow_cc.so.2
#3  0x00001539f688b728 in std::_Function_handler<void (), tsl::thread::EigenEnvironment::CreateThread(std::function<void ()>)::{lambda()#1}>::_M_invoke(std::_Any_data const&) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week1/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_MULTIARCHS_X_2024-08-18-0000/external/el8_amd64_gcc12/lib/scram_x86-64-v3/libtensorflow_cc.so.2
#4  0x00001539e569cbf2 in tsl::(anonymous namespace)::PThread::ThreadFn(void*) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week1/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_MULTIARCHS_X_2024-08-18-0000/external/el8_amd64_gcc12/lib/scram_x86-64-v3/libtensorflow_framework.so.2
#5  0x0000153a320661ca in start_thread () from /lib64/libpthread.so.0
#6  0x0000153a31cc18d3 in clone () from /lib64/libc.so.6
Thread 25 (Thread 0x1539aafff700 (LWP 3435628) "cmsRun"):
#0  0x0000153a3206c48c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00001539f688d9a6 in Eigen::ThreadPoolTempl<tsl::thread::EigenEnvironment>::WaitForWork(Eigen::EventCount::Waiter*, tsl::thread::EigenEnvironment::Task*) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week1/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_MULTIARCHS_X_2024-08-18-0000/external/el8_amd64_gcc12/lib/scram_x86-64-v3/libtensorflow_cc.so.2
#2  0x00001539f688dfa5 in Eigen::ThreadPoolTempl<tsl::thread::EigenEnvironment>::WorkerLoop(int) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week1/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_MULTIARCHS_X_2024-08-18-0000/external/el8_amd64_gcc12/lib/scram_x86-64-v3/libtensorflow_cc.so.2
#3  0x00001539f688b728 in std::_Function_handler<void (), tsl::thread::EigenEnvironment::CreateThread(std::function<void ()>)::{lambda()#1}>::_M_invoke(std::_Any_data const&) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week1/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_MULTIARCHS_X_2024-08-18-0000/external/el8_amd64_gcc12/lib/scram_x86-64-v3/libtensorflow_cc.so.2
#4  0x00001539e569cbf2 in tsl::(anonymous namespace)::PThread::ThreadFn(void*) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week1/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_MULTIARCHS_X_2024-08-18-0000/external/el8_amd64_gcc12/lib/scram_x86-64-v3/libtensorflow_framework.so.2
#5  0x0000153a320661ca in start_thread () from /lib64/libpthread.so.0
#6  0x0000153a31cc18d3 in clone () from /lib64/libc.so.6
Thread 24 (Thread 0x1539ab524700 (LWP 3435627) "cmsRun"):
#0  0x0000153a3206c48c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00001539f688d9a6 in Eigen::ThreadPoolTempl<tsl::thread::EigenEnvironment>::WaitForWork(Eigen::EventCount::Waiter*, tsl::thread::EigenEnvironment::Task*) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week1/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_MULTIARCHS_X_2024-08-18-0000/external/el8_amd64_gcc12/lib/scram_x86-64-v3/libtensorflow_cc.so.2
#2  0x00001539f688dfa5 in Eigen::ThreadPoolTempl<tsl::thread::EigenEnvironment>::WorkerLoop(int) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week1/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_MULTIARCHS_X_2024-08-18-0000/external/el8_amd64_gcc12/lib/scram_x86-64-v3/libtensorflow_cc.so.2
#3  0x00001539f688b728 in std::_Function_handler<void (), tsl::thread::EigenEnvironment::CreateThread(std::function<void ()>)::{lambda()#1}>::_M_invoke(std::_Any_data const&) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week1/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_MULTIARCHS_X_2024-08-18-0000/external/el8_amd64_gcc12/lib/scram_x86-64-v3/libtensorflow_cc.so.2
#4  0x00001539e569cbf2 in tsl::(anonymous namespace)::PThread::ThreadFn(void*) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week1/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_MULTIARCHS_X_2024-08-18-0000/external/el8_amd64_gcc12/lib/scram_x86-64-v3/libtensorflow_framework.so.2
#5  0x0000153a320661ca in start_thread () from /lib64/libpthread.so.0
#6  0x0000153a31cc18d3 in clone () from /lib64/libc.so.6
Thread 23 (Thread 0x1539d7bf2700 (LWP 3435618) "cmsRun"):
#0  0x0000153a3206edd6 in do_futex_wait.constprop () from /lib64/libpthread.so.0
#1  0x0000153a3206eec8 in __new_sem_wait_slow.constprop.0 () from /lib64/libpthread.so.0
#2  0x0000153a285ce7c6 in XrdCl::JobManager::RunJobs() () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week1/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_MULTIARCHS_X_2024-08-18-0000/external/el8_amd64_gcc12/lib/libXrdCl.so.3
#3  0x0000153a285ce879 in RunRunnerThread () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week1/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_MULTIARCHS_X_2024-08-18-0000/external/el8_amd64_gcc12/lib/libXrdCl.so.3
#4  0x0000153a320661ca in start_thread () from /lib64/libpthread.so.0
#5  0x0000153a31cc18d3 in clone () from /lib64/libc.so.6
Thread 22 (Thread 0x1539d7df3700 (LWP 3435617) "cmsRun"):
#0  0x0000153a3206edd6 in do_futex_wait.constprop () from /lib64/libpthread.so.0
#1  0x0000153a3206eec8 in __new_sem_wait_slow.constprop.0 () from /lib64/libpthread.so.0
#2  0x0000153a285ce7c6 in XrdCl::JobManager::RunJobs() () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week1/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_MULTIARCHS_X_2024-08-18-0000/external/el8_amd64_gcc12/lib/libXrdCl.so.3
#3  0x0000153a285ce879 in RunRunnerThread () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week1/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_MULTIARCHS_X_2024-08-18-0000/external/el8_amd64_gcc12/lib/libXrdCl.so.3
#4  0x0000153a320661ca in start_thread () from /lib64/libpthread.so.0
#5  0x0000153a31cc18d3 in clone () from /lib64/libc.so.6
Thread 21 (Thread 0x1539d7ff4700 (LWP 3435616) "cmsRun"):
#0  0x0000153a3206edd6 in do_futex_wait.constprop () from /lib64/libpthread.so.0
#1  0x0000153a3206eec8 in __new_sem_wait_slow.constprop.0 () from /lib64/libpthread.so.0
#2  0x0000153a285ce7c6 in XrdCl::JobManager::RunJobs() () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week1/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_MULTIARCHS_X_2024-08-18-0000/external/el8_amd64_gcc12/lib/libXrdCl.so.3
#3  0x0000153a285ce879 in RunRunnerThread () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week1/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_MULTIARCHS_X_2024-08-18-0000/external/el8_amd64_gcc12/lib/libXrdCl.so.3
#4  0x0000153a320661ca in start_thread () from /lib64/libpthread.so.0
#5  0x0000153a31cc18d3 in clone () from /lib64/libc.so.6
Thread 20 (Thread 0x1539d81f5700 (LWP 3435615) "cmsRun"):
#0  0x0000153a320701b0 in nanosleep () from /lib64/libpthread.so.0
#1  0x0000153a2a7bd7e8 in XrdSysTimer::Wait(int) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week1/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_MULTIARCHS_X_2024-08-18-0000/external/el8_amd64_gcc12/lib/libXrdUtils.so.3
#2  0x0000153a2853f37c in XrdCl::TaskManager::RunTasks() () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week1/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_MULTIARCHS_X_2024-08-18-0000/external/el8_amd64_gcc12/lib/libXrdCl.so.3
#3  0x0000153a2853f4b9 in RunRunnerThread () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week1/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_MULTIARCHS_X_2024-08-18-0000/external/el8_amd64_gcc12/lib/libXrdCl.so.3
#4  0x0000153a320661ca in start_thread () from /lib64/libpthread.so.0
#5  0x0000153a31cc18d3 in clone () from /lib64/libc.so.6
Thread 19 (Thread 0x1539d83f6700 (LWP 3435614) "cmsRun"):
#0  0x0000153a31dc7307 in epoll_wait () from /lib64/libc.so.6
#1  0x0000153a2a7b7b62 in XrdSys::IOEvents::PollE::Begin(XrdSysSemaphore*, int&, char const**) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week1/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_MULTIARCHS_X_2024-08-18-0000/external/el8_amd64_gcc12/lib/libXrdUtils.so.3
#2  0x0000153a2a7b3acd in XrdSys::IOEvents::BootStrap::Start(void*) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week1/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_MULTIARCHS_X_2024-08-18-0000/external/el8_amd64_gcc12/lib/libXrdUtils.so.3
#3  0x0000153a2a7bcef7 in XrdSysThread_Xeq () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week1/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_MULTIARCHS_X_2024-08-18-0000/external/el8_amd64_gcc12/lib/libXrdUtils.so.3
#4  0x0000153a320661ca in start_thread () from /lib64/libpthread.so.0
#5  0x0000153a31cc18d3 in clone () from /lib64/libc.so.6
Thread 18 (Thread 0x1539d85f7700 (LWP 3435613) "cmsRun"):
#0  0x0000153a31dc7307 in epoll_wait () from /lib64/libc.so.6
#1  0x0000153a2a7b7b62 in XrdSys::IOEvents::PollE::Begin(XrdSysSemaphore*, int&, char const**) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week1/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_MULTIARCHS_X_2024-08-18-0000/external/el8_amd64_gcc12/lib/libXrdUtils.so.3
#2  0x0000153a2a7b3acd in XrdSys::IOEvents::BootStrap::Start(void*) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week1/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_MULTIARCHS_X_2024-08-18-0000/external/el8_amd64_gcc12/lib/libXrdUtils.so.3
#3  0x0000153a2a7bcef7 in XrdSysThread_Xeq () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week1/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_MULTIARCHS_X_2024-08-18-0000/external/el8_amd64_gcc12/lib/libXrdUtils.so.3
#4  0x0000153a320661ca in start_thread () from /lib64/libpthread.so.0
#5  0x0000153a31cc18d3 in clone () from /lib64/libc.so.6
Thread 17 (Thread 0x1539d87f8700 (LWP 3435612) "cmsRun"):
#0  0x0000153a31dc7307 in epoll_wait () from /lib64/libc.so.6
#1  0x0000153a2a7b7b62 in XrdSys::IOEvents::PollE::Begin(XrdSysSemaphore*, int&, char const**) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week1/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_MULTIARCHS_X_2024-08-18-0000/external/el8_amd64_gcc12/lib/libXrdUtils.so.3
#2  0x0000153a2a7b3acd in XrdSys::IOEvents::BootStrap::Start(void*) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week1/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_MULTIARCHS_X_2024-08-18-0000/external/el8_amd64_gcc12/lib/libXrdUtils.so.3
#3  0x0000153a2a7bcef7 in XrdSysThread_Xeq () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week1/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_MULTIARCHS_X_2024-08-18-0000/external/el8_amd64_gcc12/lib/libXrdUtils.so.3
#4  0x0000153a320661ca in start_thread () from /lib64/libpthread.so.0
#5  0x0000153a31cc18d3 in clone () from /lib64/libc.so.6
Thread 16 (Thread 0x1539d89f9700 (LWP 3435611) "cmsRun"):
#0  0x0000153a31dc7307 in epoll_wait () from /lib64/libc.so.6
#1  0x0000153a2a7b7b62 in XrdSys::IOEvents::PollE::Begin(XrdSysSemaphore*, int&, char const**) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week1/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_MULTIARCHS_X_2024-08-18-0000/external/el8_amd64_gcc12/lib/libXrdUtils.so.3
#2  0x0000153a2a7b3acd in XrdSys::IOEvents::BootStrap::Start(void*) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week1/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_MULTIARCHS_X_2024-08-18-0000/external/el8_amd64_gcc12/lib/libXrdUtils.so.3
#3  0x0000153a2a7bcef7 in XrdSysThread_Xeq () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week1/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_MULTIARCHS_X_2024-08-18-0000/external/el8_amd64_gcc12/lib/libXrdUtils.so.3
#4  0x0000153a320661ca in start_thread () from /lib64/libpthread.so.0
#5  0x0000153a31cc18d3 in clone () from /lib64/libc.so.6
Thread 15 (Thread 0x1539d8bfa700 (LWP 3435610) "cmsRun"):
#0  0x0000153a31dc7307 in epoll_wait () from /lib64/libc.so.6
#1  0x0000153a2a7b7b62 in XrdSys::IOEvents::PollE::Begin(XrdSysSemaphore*, int&, char const**) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week1/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_MULTIARCHS_X_2024-08-18-0000/external/el8_amd64_gcc12/lib/libXrdUtils.so.3
#2  0x0000153a2a7b3acd in XrdSys::IOEvents::BootStrap::Start(void*) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week1/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_MULTIARCHS_X_2024-08-18-0000/external/el8_amd64_gcc12/lib/libXrdUtils.so.3
#3  0x0000153a2a7bcef7 in XrdSysThread_Xeq () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week1/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_MULTIARCHS_X_2024-08-18-0000/external/el8_amd64_gcc12/lib/libXrdUtils.so.3
#4  0x0000153a320661ca in start_thread () from /lib64/libpthread.so.0
#5  0x0000153a31cc18d3 in clone () from /lib64/libc.so.6
Thread 14 (Thread 0x1539d8dfb700 (LWP 3435609) "cmsRun"):
#0  0x0000153a31dc7307 in epoll_wait () from /lib64/libc.so.6
#1  0x0000153a2a7b7b62 in XrdSys::IOEvents::PollE::Begin(XrdSysSemaphore*, int&, char const**) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week1/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_MULTIARCHS_X_2024-08-18-0000/external/el8_amd64_gcc12/lib/libXrdUtils.so.3
#2  0x0000153a2a7b3acd in XrdSys::IOEvents::BootStrap::Start(void*) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week1/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_MULTIARCHS_X_2024-08-18-0000/external/el8_amd64_gcc12/lib/libXrdUtils.so.3
#3  0x0000153a2a7bcef7 in XrdSysThread_Xeq () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week1/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_MULTIARCHS_X_2024-08-18-0000/external/el8_amd64_gcc12/lib/libXrdUtils.so.3
#4  0x0000153a320661ca in start_thread () from /lib64/libpthread.so.0
#5  0x0000153a31cc18d3 in clone () from /lib64/libc.so.6
Thread 13 (Thread 0x1539d8ffc700 (LWP 3435608) "cmsRun"):
#0  0x0000153a31dc7307 in epoll_wait () from /lib64/libc.so.6
#1  0x0000153a2a7b7b62 in XrdSys::IOEvents::PollE::Begin(XrdSysSemaphore*, int&, char const**) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week1/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_MULTIARCHS_X_2024-08-18-0000/external/el8_amd64_gcc12/lib/libXrdUtils.so.3
#2  0x0000153a2a7b3acd in XrdSys::IOEvents::BootStrap::Start(void*) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week1/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_MULTIARCHS_X_2024-08-18-0000/external/el8_amd64_gcc12/lib/libXrdUtils.so.3
#3  0x0000153a2a7bcef7 in XrdSysThread_Xeq () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week1/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_MULTIARCHS_X_2024-08-18-0000/external/el8_amd64_gcc12/lib/libXrdUtils.so.3
#4  0x0000153a320661ca in start_thread () from /lib64/libpthread.so.0
#5  0x0000153a31cc18d3 in clone () from /lib64/libc.so.6
Thread 12 (Thread 0x1539d91fd700 (LWP 3435607) "cmsRun"):
#0  0x0000153a31dc7307 in epoll_wait () from /lib64/libc.so.6
#1  0x0000153a2a7b7b62 in XrdSys::IOEvents::PollE::Begin(XrdSysSemaphore*, int&, char const**) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week1/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_MULTIARCHS_X_2024-08-18-0000/external/el8_amd64_gcc12/lib/libXrdUtils.so.3
#2  0x0000153a2a7b3acd in XrdSys::IOEvents::BootStrap::Start(void*) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week1/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_MULTIARCHS_X_2024-08-18-0000/external/el8_amd64_gcc12/lib/libXrdUtils.so.3
#3  0x0000153a2a7bcef7 in XrdSysThread_Xeq () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week1/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_MULTIARCHS_X_2024-08-18-0000/external/el8_amd64_gcc12/lib/libXrdUtils.so.3
#4  0x0000153a320661ca in start_thread () from /lib64/libpthread.so.0
#5  0x0000153a31cc18d3 in clone () from /lib64/libc.so.6
Thread 11 (Thread 0x1539d93fe700 (LWP 3435606) "cmsRun"):
#0  0x0000153a31dc7307 in epoll_wait () from /lib64/libc.so.6
#1  0x0000153a2a7b7b62 in XrdSys::IOEvents::PollE::Begin(XrdSysSemaphore*, int&, char const**) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week1/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_MULTIARCHS_X_2024-08-18-0000/external/el8_amd64_gcc12/lib/libXrdUtils.so.3
#2  0x0000153a2a7b3acd in XrdSys::IOEvents::BootStrap::Start(void*) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week1/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_MULTIARCHS_X_2024-08-18-0000/external/el8_amd64_gcc12/lib/libXrdUtils.so.3
#3  0x0000153a2a7bcef7 in XrdSysThread_Xeq () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week1/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_MULTIARCHS_X_2024-08-18-0000/external/el8_amd64_gcc12/lib/libXrdUtils.so.3
#4  0x0000153a320661ca in start_thread () from /lib64/libpthread.so.0
#5  0x0000153a31cc18d3 in clone () from /lib64/libc.so.6
Thread 10 (Thread 0x1539d95ff700 (LWP 3435605) "cmsRun"):
#0  0x0000153a31dc7307 in epoll_wait () from /lib64/libc.so.6
#1  0x0000153a2a7b7b62 in XrdSys::IOEvents::PollE::Begin(XrdSysSemaphore*, int&, char const**) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week1/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_MULTIARCHS_X_2024-08-18-0000/external/el8_amd64_gcc12/lib/libXrdUtils.so.3
#2  0x0000153a2a7b3acd in XrdSys::IOEvents::BootStrap::Start(void*) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week1/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_MULTIARCHS_X_2024-08-18-0000/external/el8_amd64_gcc12/lib/libXrdUtils.so.3
#3  0x0000153a2a7bcef7 in XrdSysThread_Xeq () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week1/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_MULTIARCHS_X_2024-08-18-0000/external/el8_amd64_gcc12/lib/libXrdUtils.so.3
#4  0x0000153a320661ca in start_thread () from /lib64/libpthread.so.0
#5  0x0000153a31cc18d3 in clone () from /lib64/libc.so.6
Thread 9 (Thread 0x1539da5ff700 (LWP 3435604) "cmsRun"):
#0  0x0000153a31d90098 in nanosleep () from /lib64/libc.so.6
#1  0x0000153a31d8ff9e in sleep () from /lib64/libc.so.6
#2  0x0000153a2b171f90 in sig_pause_for_stacktrace () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week1/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_MULTIARCHS_X_2024-08-18-0000/lib/el8_amd64_gcc12/scram_x86-64-v3/pluginFWCoreServicesPlugins.so
#3  <signal handler called>
#4  0x0000153a34964a0e in edm::StreamSchedule::processOneStreamAsync<edm::OccurrenceTraits<edm::RunPrincipal, (edm::BranchActionType)2> >(edm::WaitingTaskHolder, edm::OccurrenceTraits<edm::RunPrincipal, (edm::BranchActionType)2>::TransitionInfoType&, edm::ServiceToken const&, bool)::{lambda()#2}::operator()() () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week1/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_MULTIARCHS_X_2024-08-18-0000/lib/el8_amd64_gcc12/scram_x86-64-v3/libFWCoreFramework.so
#5  0x0000153a34974127 in tbb::detail::d1::enqueue_task<edm::StreamSchedule::processOneStreamAsync<edm::OccurrenceTraits<edm::RunPrincipal, (edm::BranchActionType)2> >(edm::WaitingTaskHolder, edm::OccurrenceTraits<edm::RunPrincipal, (edm::BranchActionType)2>::TransitionInfoType&, edm::ServiceToken const&, bool)::{lambda()#4}>::execute(tbb::detail::d1::execution_data&) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week1/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_MULTIARCHS_X_2024-08-18-0000/lib/el8_amd64_gcc12/scram_x86-64-v3/libFWCoreFramework.so
#6  0x0000153a33d07b3b in tbb::detail::r1::task_dispatcher::local_wait_for_all<false, tbb::detail::r1::outermost_worker_waiter> (t=0x153996721d00, waiter=..., this=0x153a307bbc80) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-2391c941213c757dc9a1835b31681235/tbb-v2021.9.0/src/tbb/task_dispatcher.h:322
#7  tbb::detail::r1::task_dispatcher::local_wait_for_all<tbb::detail::r1::outermost_worker_waiter> (t=0x0, waiter=..., this=0x153a307bbc80) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-2391c941213c757dc9a1835b31681235/tbb-v2021.9.0/src/tbb/task_dispatcher.h:458
#8  tbb::detail::r1::arena::process (tls=..., this=<optimized out>) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-2391c941213c757dc9a1835b31681235/tbb-v2021.9.0/src/tbb/arena.cpp:137
#9  tbb::detail::r1::market::process (this=<optimized out>, j=...) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-2391c941213c757dc9a1835b31681235/tbb-v2021.9.0/src/tbb/market.cpp:599
#10 0x0000153a33d09cee in tbb::detail::r1::rml::private_worker::run (this=0x153a2b766e00) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-2391c941213c757dc9a1835b31681235/tbb-v2021.9.0/src/tbb/private_server.cpp:271
#11 tbb::detail::r1::rml::private_worker::thread_routine (arg=0x153a2b766e00) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-2391c941213c757dc9a1835b31681235/tbb-v2021.9.0/src/tbb/private_server.cpp:221
#12 0x0000153a320661ca in start_thread () from /lib64/libpthread.so.0
#13 0x0000153a31cc18d3 in clone () from /lib64/libc.so.6
Thread 8 (Thread 0x1539db5fc700 (LWP 3435603) "cmsRun"):
#0  0x0000153a31d90098 in nanosleep () from /lib64/libc.so.6
#1  0x0000153a31d8ff9e in sleep () from /lib64/libc.so.6
#2  0x0000153a2b171f90 in sig_pause_for_stacktrace () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week1/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_MULTIARCHS_X_2024-08-18-0000/lib/el8_amd64_gcc12/scram_x86-64-v3/pluginFWCoreServicesPlugins.so
#3  <signal handler called>
#4  0x0000153a34964285 in edm::StreamSchedule::processOneStreamAsync<edm::OccurrenceTraits<edm::RunPrincipal, (edm::BranchActionType)2> >(edm::WaitingTaskHolder, edm::OccurrenceTraits<edm::RunPrincipal, (edm::BranchActionType)2>::TransitionInfoType&, edm::ServiceToken const&, bool)::{lambda()#2}::operator()() () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week1/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_MULTIARCHS_X_2024-08-18-0000/lib/el8_amd64_gcc12/scram_x86-64-v3/libFWCoreFramework.so
#5  0x0000153a3497419d in tbb::detail::d1::function_task<edm::StreamSchedule::processOneStreamAsync<edm::OccurrenceTraits<edm::RunPrincipal, (edm::BranchActionType)2> >(edm::WaitingTaskHolder, edm::OccurrenceTraits<edm::RunPrincipal, (edm::BranchActionType)2>::TransitionInfoType&, edm::ServiceToken const&, bool)::{lambda()#3}>::execute(tbb::detail::d1::execution_data&) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week1/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_MULTIARCHS_X_2024-08-18-0000/lib/el8_amd64_gcc12/scram_x86-64-v3/libFWCoreFramework.so
#6  0x0000153a33d07b3b in tbb::detail::r1::task_dispatcher::local_wait_for_all<false, tbb::detail::r1::outermost_worker_waiter> (t=0x1539966e3200, waiter=..., this=0x153a307bbb80) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-2391c941213c757dc9a1835b31681235/tbb-v2021.9.0/src/tbb/task_dispatcher.h:322
#7  tbb::detail::r1::task_dispatcher::local_wait_for_all<tbb::detail::r1::outermost_worker_waiter> (t=0x0, waiter=..., this=0x153a307bbb80) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-2391c941213c757dc9a1835b31681235/tbb-v2021.9.0/src/tbb/task_dispatcher.h:458
#8  tbb::detail::r1::arena::process (tls=..., this=<optimized out>) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-2391c941213c757dc9a1835b31681235/tbb-v2021.9.0/src/tbb/arena.cpp:137
#9  tbb::detail::r1::market::process (this=<optimized out>, j=...) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-2391c941213c757dc9a1835b31681235/tbb-v2021.9.0/src/tbb/market.cpp:599
#10 0x0000153a33d09cee in tbb::detail::r1::rml::private_worker::run (this=0x153a2b766f00) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-2391c941213c757dc9a1835b31681235/tbb-v2021.9.0/src/tbb/private_server.cpp:271
#11 tbb::detail::r1::rml::private_worker::thread_routine (arg=0x153a2b766f00) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-2391c941213c757dc9a1835b31681235/tbb-v2021.9.0/src/tbb/private_server.cpp:221
#12 0x0000153a320661ca in start_thread () from /lib64/libpthread.so.0
#13 0x0000153a31cc18d3 in clone () from /lib64/libc.so.6
Thread 7 (Thread 0x1539dbffd700 (LWP 3435602) "cmsRun"):
#0  0x0000153a31d90098 in nanosleep () from /lib64/libc.so.6
#1  0x0000153a31d8ff9e in sleep () from /lib64/libc.so.6
#2  0x0000153a2b171f90 in sig_pause_for_stacktrace () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week1/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_MULTIARCHS_X_2024-08-18-0000/lib/el8_amd64_gcc12/scram_x86-64-v3/pluginFWCoreServicesPlugins.so
#3  <signal handler called>
#4  0x0000153a349642ae in edm::StreamSchedule::processOneStreamAsync<edm::OccurrenceTraits<edm::RunPrincipal, (edm::BranchActionType)2> >(edm::WaitingTaskHolder, edm::OccurrenceTraits<edm::RunPrincipal, (edm::BranchActionType)2>::TransitionInfoType&, edm::ServiceToken const&, bool)::{lambda()#2}::operator()() () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week1/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_MULTIARCHS_X_2024-08-18-0000/lib/el8_amd64_gcc12/scram_x86-64-v3/libFWCoreFramework.so
#5  0x0000153a34974127 in tbb::detail::d1::enqueue_task<edm::StreamSchedule::processOneStreamAsync<edm::OccurrenceTraits<edm::RunPrincipal, (edm::BranchActionType)2> >(edm::WaitingTaskHolder, edm::OccurrenceTraits<edm::RunPrincipal, (edm::BranchActionType)2>::TransitionInfoType&, edm::ServiceToken const&, bool)::{lambda()#4}>::execute(tbb::detail::d1::execution_data&) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week1/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_MULTIARCHS_X_2024-08-18-0000/lib/el8_amd64_gcc12/scram_x86-64-v3/libFWCoreFramework.so
#6  0x0000153a33d07b3b in tbb::detail::r1::task_dispatcher::local_wait_for_all<false, tbb::detail::r1::outermost_worker_waiter> (t=0x1539968b0a00, waiter=..., this=0x153a307bba80) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-2391c941213c757dc9a1835b31681235/tbb-v2021.9.0/src/tbb/task_dispatcher.h:322
#7  tbb::detail::r1::task_dispatcher::local_wait_for_all<tbb::detail::r1::outermost_worker_waiter> (t=0x0, waiter=..., this=0x153a307bba80) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-2391c941213c757dc9a1835b31681235/tbb-v2021.9.0/src/tbb/task_dispatcher.h:458
#8  tbb::detail::r1::arena::process (tls=..., this=<optimized out>) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-2391c941213c757dc9a1835b31681235/tbb-v2021.9.0/src/tbb/arena.cpp:137
#9  tbb::detail::r1::market::process (this=<optimized out>, j=...) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-2391c941213c757dc9a1835b31681235/tbb-v2021.9.0/src/tbb/market.cpp:599
#10 0x0000153a33d09cee in tbb::detail::r1::rml::private_worker::run (this=0x153a2b767000) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-2391c941213c757dc9a1835b31681235/tbb-v2021.9.0/src/tbb/private_server.cpp:271
#11 tbb::detail::r1::rml::private_worker::thread_routine (arg=0x153a2b767000) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-2391c941213c757dc9a1835b31681235/tbb-v2021.9.0/src/tbb/private_server.cpp:221
#12 0x0000153a320661ca in start_thread () from /lib64/libpthread.so.0
#13 0x0000153a31cc18d3 in clone () from /lib64/libc.so.6
Thread 6 (Thread 0x1539dd3ff700 (LWP 3435601) "cmsRun"):
#0  0x0000153a31d90098 in nanosleep () from /lib64/libc.so.6
#1  0x0000153a31d8ff9e in sleep () from /lib64/libc.so.6
#2  0x0000153a2b171f90 in sig_pause_for_stacktrace () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week1/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_MULTIARCHS_X_2024-08-18-0000/lib/el8_amd64_gcc12/scram_x86-64-v3/pluginFWCoreServicesPlugins.so
#3  <signal handler called>
#4  0x0000153a34961fed in std::__weak_ptr<edm::serviceregistry::ServicesManager, (__gnu_cxx::_Lock_policy)2>::__weak_ptr(std::__weak_ptr<edm::serviceregistry::ServicesManager, (__gnu_cxx::_Lock_policy)2> const&) [clone .lto_priv.0] () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week1/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_MULTIARCHS_X_2024-08-18-0000/lib/el8_amd64_gcc12/scram_x86-64-v3/libFWCoreFramework.so
#5  0x0000153a34964ab5 in edm::StreamSchedule::processOneStreamAsync<edm::OccurrenceTraits<edm::RunPrincipal, (edm::BranchActionType)2> >(edm::WaitingTaskHolder, edm::OccurrenceTraits<edm::RunPrincipal, (edm::BranchActionType)2>::TransitionInfoType&, edm::ServiceToken const&, bool)::{lambda()#2}::operator()() () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week1/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_MULTIARCHS_X_2024-08-18-0000/lib/el8_amd64_gcc12/scram_x86-64-v3/libFWCoreFramework.so
#6  0x0000153a34974127 in tbb::detail::d1::enqueue_task<edm::StreamSchedule::processOneStreamAsync<edm::OccurrenceTraits<edm::RunPrincipal, (edm::BranchActionType)2> >(edm::WaitingTaskHolder, edm::OccurrenceTraits<edm::RunPrincipal, (edm::BranchActionType)2>::TransitionInfoType&, edm::ServiceToken const&, bool)::{lambda()#4}>::execute(tbb::detail::d1::execution_data&) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week1/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_MULTIARCHS_X_2024-08-18-0000/lib/el8_amd64_gcc12/scram_x86-64-v3/libFWCoreFramework.so
#7  0x0000153a33d07b3b in tbb::detail::r1::task_dispatcher::local_wait_for_all<false, tbb::detail::r1::outermost_worker_waiter> (t=0x153996860c00, waiter=..., this=0x153a307bb980) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-2391c941213c757dc9a1835b31681235/tbb-v2021.9.0/src/tbb/task_dispatcher.h:322
#8  tbb::detail::r1::task_dispatcher::local_wait_for_all<tbb::detail::r1::outermost_worker_waiter> (t=0x0, waiter=..., this=0x153a307bb980) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-2391c941213c757dc9a1835b31681235/tbb-v2021.9.0/src/tbb/task_dispatcher.h:458
#9  tbb::detail::r1::arena::process (tls=..., this=<optimized out>) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-2391c941213c757dc9a1835b31681235/tbb-v2021.9.0/src/tbb/arena.cpp:137
#10 tbb::detail::r1::market::process (this=<optimized out>, j=...) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-2391c941213c757dc9a1835b31681235/tbb-v2021.9.0/src/tbb/market.cpp:599
#11 0x0000153a33d09cee in tbb::detail::r1::rml::private_worker::run (this=0x153a2b766e80) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-2391c941213c757dc9a1835b31681235/tbb-v2021.9.0/src/tbb/private_server.cpp:271
#12 tbb::detail::r1::rml::private_worker::thread_routine (arg=0x153a2b766e80) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-2391c941213c757dc9a1835b31681235/tbb-v2021.9.0/src/tbb/private_server.cpp:221
#13 0x0000153a320661ca in start_thread () from /lib64/libpthread.so.0
#14 0x0000153a31cc18d3 in clone () from /lib64/libc.so.6
Thread 5 (Thread 0x1539dc9fe700 (LWP 3435600) "cmsRun"):
#0  0x0000153a31d90098 in nanosleep () from /lib64/libc.so.6
#1  0x0000153a31d8ff9e in sleep () from /lib64/libc.so.6
#2  0x0000153a2b171f90 in sig_pause_for_stacktrace () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week1/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_MULTIARCHS_X_2024-08-18-0000/lib/el8_amd64_gcc12/scram_x86-64-v3/pluginFWCoreServicesPlugins.so
#3  <signal handler called>
#4  0x0000153a31cc141d in syscall () from /lib64/libc.so.6
#5  0x0000153a33d09fd2 in tbb::detail::r1::futex_wait (comparand=2, futex=0x153a2b766fa4) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-2391c941213c757dc9a1835b31681235/tbb-v2021.9.0/src/tbb/semaphore.h:100
#6  tbb::detail::r1::binary_semaphore::P (this=0x153a2b766fa4) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-2391c941213c757dc9a1835b31681235/tbb-v2021.9.0/src/tbb/semaphore.h:253
#7  tbb::detail::r1::rml::internal::thread_monitor::wait (this=0x153a2b766fa0) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-2391c941213c757dc9a1835b31681235/tbb-v2021.9.0/src/tbb/rml_thread_monitor.h:235
#8  tbb::detail::r1::rml::private_worker::run (this=0x153a2b766f80) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-2391c941213c757dc9a1835b31681235/tbb-v2021.9.0/src/tbb/private_server.cpp:273
#9  tbb::detail::r1::rml::private_worker::thread_routine (arg=0x153a2b766f80) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-2391c941213c757dc9a1835b31681235/tbb-v2021.9.0/src/tbb/private_server.cpp:221
#10 0x0000153a320661ca in start_thread () from /lib64/libpthread.so.0
#11 0x0000153a31cc18d3 in clone () from /lib64/libc.so.6
Thread 4 (Thread 0x1539de7c7700 (LWP 3435599) "cmsRun"):
#0  0x0000153a31d90098 in nanosleep () from /lib64/libc.so.6
#1  0x0000153a31d8ff9e in sleep () from /lib64/libc.so.6
#2  0x0000153a2b171f90 in sig_pause_for_stacktrace () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week1/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_MULTIARCHS_X_2024-08-18-0000/lib/el8_amd64_gcc12/scram_x86-64-v3/pluginFWCoreServicesPlugins.so
#3  <signal handler called>
#4  0x0000153a348f1c10 in tbb::detail::r1::spawn(tbb::detail::d1::task&, tbb::detail::d1::task_group_context&)@plt () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week1/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_MULTIARCHS_X_2024-08-18-0000/lib/el8_amd64_gcc12/scram_x86-64-v3/libFWCoreFramework.so
#5  0x0000153a34964ade in edm::StreamSchedule::processOneStreamAsync<edm::OccurrenceTraits<edm::RunPrincipal, (edm::BranchActionType)2> >(edm::WaitingTaskHolder, edm::OccurrenceTraits<edm::RunPrincipal, (edm::BranchActionType)2>::TransitionInfoType&, edm::ServiceToken const&, bool)::{lambda()#2}::operator()() () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week1/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_MULTIARCHS_X_2024-08-18-0000/lib/el8_amd64_gcc12/scram_x86-64-v3/libFWCoreFramework.so
#6  0x0000153a34974127 in tbb::detail::d1::enqueue_task<edm::StreamSchedule::processOneStreamAsync<edm::OccurrenceTraits<edm::RunPrincipal, (edm::BranchActionType)2> >(edm::WaitingTaskHolder, edm::OccurrenceTraits<edm::RunPrincipal, (edm::BranchActionType)2>::TransitionInfoType&, edm::ServiceToken const&, bool)::{lambda()#4}>::execute(tbb::detail::d1::execution_data&) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week1/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_MULTIARCHS_X_2024-08-18-0000/lib/el8_amd64_gcc12/scram_x86-64-v3/libFWCoreFramework.so
#7  0x0000153a33d07b3b in tbb::detail::r1::task_dispatcher::local_wait_for_all<false, tbb::detail::r1::outermost_worker_waiter> (t=0x153996862100, waiter=..., this=0x153a307bbb00) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-2391c941213c757dc9a1835b31681235/tbb-v2021.9.0/src/tbb/task_dispatcher.h:322
#8  tbb::detail::r1::task_dispatcher::local_wait_for_all<tbb::detail::r1::outermost_worker_waiter> (t=0x0, waiter=..., this=0x153a307bbb00) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-2391c941213c757dc9a1835b31681235/tbb-v2021.9.0/src/tbb/task_dispatcher.h:458
#9  tbb::detail::r1::arena::process (tls=..., this=<optimized out>) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-2391c941213c757dc9a1835b31681235/tbb-v2021.9.0/src/tbb/arena.cpp:137
#10 tbb::detail::r1::market::process (this=<optimized out>, j=...) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-2391c941213c757dc9a1835b31681235/tbb-v2021.9.0/src/tbb/market.cpp:599
#11 0x0000153a33d09cee in tbb::detail::r1::rml::private_worker::run (this=0x153a2b767100) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-2391c941213c757dc9a1835b31681235/tbb-v2021.9.0/src/tbb/private_server.cpp:271
#12 tbb::detail::r1::rml::private_worker::thread_routine (arg=0x153a2b767100) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-2391c941213c757dc9a1835b31681235/tbb-v2021.9.0/src/tbb/private_server.cpp:221
#13 0x0000153a320661ca in start_thread () from /lib64/libpthread.so.0
#14 0x0000153a31cc18d3 in clone () from /lib64/libc.so.6
Thread 3 (Thread 0x1539df1c8700 (LWP 3435598) "cmsRun"):
#0  0x0000153a31d90098 in nanosleep () from /lib64/libc.so.6
#1  0x0000153a31d8ff9e in sleep () from /lib64/libc.so.6
#2  0x0000153a2b171f90 in sig_pause_for_stacktrace () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week1/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_MULTIARCHS_X_2024-08-18-0000/lib/el8_amd64_gcc12/scram_x86-64-v3/pluginFWCoreServicesPlugins.so
#3  <signal handler called>
#4  0x0000153a34964285 in edm::StreamSchedule::processOneStreamAsync<edm::OccurrenceTraits<edm::RunPrincipal, (edm::BranchActionType)2> >(edm::WaitingTaskHolder, edm::OccurrenceTraits<edm::RunPrincipal, (edm::BranchActionType)2>::TransitionInfoType&, edm::ServiceToken const&, bool)::{lambda()#2}::operator()() () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week1/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_MULTIARCHS_X_2024-08-18-0000/lib/el8_amd64_gcc12/scram_x86-64-v3/libFWCoreFramework.so
#5  0x0000153a34974127 in tbb::detail::d1::enqueue_task<edm::StreamSchedule::processOneStreamAsync<edm::OccurrenceTraits<edm::RunPrincipal, (edm::BranchActionType)2> >(edm::WaitingTaskHolder, edm::OccurrenceTraits<edm::RunPrincipal, (edm::BranchActionType)2>::TransitionInfoType&, edm::ServiceToken const&, bool)::{lambda()#4}>::execute(tbb::detail::d1::execution_data&) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week1/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_MULTIARCHS_X_2024-08-18-0000/lib/el8_amd64_gcc12/scram_x86-64-v3/libFWCoreFramework.so
#6  0x0000153a33d07b3b in tbb::detail::r1::task_dispatcher::local_wait_for_all<false, tbb::detail::r1::outermost_worker_waiter> (t=0x153996746a00, waiter=..., this=0x153a307bba00) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-2391c941213c757dc9a1835b31681235/tbb-v2021.9.0/src/tbb/task_dispatcher.h:322
#7  tbb::detail::r1::task_dispatcher::local_wait_for_all<tbb::detail::r1::outermost_worker_waiter> (t=0x0, waiter=..., this=0x153a307bba00) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-2391c941213c757dc9a1835b31681235/tbb-v2021.9.0/src/tbb/task_dispatcher.h:458
#8  tbb::detail::r1::arena::process (tls=..., this=<optimized out>) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-2391c941213c757dc9a1835b31681235/tbb-v2021.9.0/src/tbb/arena.cpp:137
#9  tbb::detail::r1::market::process (this=<optimized out>, j=...) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-2391c941213c757dc9a1835b31681235/tbb-v2021.9.0/src/tbb/market.cpp:599
#10 0x0000153a33d09cee in tbb::detail::r1::rml::private_worker::run (this=0x153a2b767080) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-2391c941213c757dc9a1835b31681235/tbb-v2021.9.0/src/tbb/private_server.cpp:271
#11 tbb::detail::r1::rml::private_worker::thread_routine (arg=0x153a2b767080) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-2391c941213c757dc9a1835b31681235/tbb-v2021.9.0/src/tbb/private_server.cpp:221
#12 0x0000153a320661ca in start_thread () from /lib64/libpthread.so.0
#13 0x0000153a31cc18d3 in clone () from /lib64/libc.so.6
Thread 2 (Thread 0x153a085a9700 (LWP 3435567) "cmsRun"):
#0  0x0000153a320706a2 in waitpid () from /lib64/libpthread.so.0
#1  0x0000153a2b1720e7 in edm::service::cmssw_stacktrace_fork() () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week1/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_MULTIARCHS_X_2024-08-18-0000/lib/el8_amd64_gcc12/scram_x86-64-v3/pluginFWCoreServicesPlugins.so
#2  0x0000153a2b1754ca in edm::service::InitRootHandlers::stacktraceHelperThread() () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week1/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_MULTIARCHS_X_2024-08-18-0000/lib/el8_amd64_gcc12/scram_x86-64-v3/pluginFWCoreServicesPlugins.so
#3  0x0000153a326d8a73 in std::execute_native_thread_routine (__p=0x153a0a5cfe80) at ../../../../../libstdc++-v3/src/c++11/thread.cc:82
#4  0x0000153a320661ca in start_thread () from /lib64/libpthread.so.0
#5  0x0000153a31cc18d3 in clone () from /lib64/libc.so.6
Thread 1 (Thread 0x153a3371c680 (LWP 3434931) "cmsRun"):
#0  0x0000153a31dbaac1 in poll () from /lib64/libc.so.6
#1  0x0000153a2b1756e7 in edm::service::InitRootHandlers::stacktraceFromThread() () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week1/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_MULTIARCHS_X_2024-08-18-0000/lib/el8_amd64_gcc12/scram_x86-64-v3/pluginFWCoreServicesPlugins.so
#2  0x0000153a2b1758e4 in sig_dostack_then_abort () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week1/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_MULTIARCHS_X_2024-08-18-0000/lib/el8_amd64_gcc12/scram_x86-64-v3/pluginFWCoreServicesPlugins.so
#3  <signal handler called>
#4  tbb::detail::r1::spawn (t=..., ctx=...) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-2391c941213c757dc9a1835b31681235/tbb-v2021.9.0/src/tbb/task_dispatcher.cpp:36
#5  0x0000153a349329f8 in tbb::detail::d1::function_task<edm::WaitingTaskHolder::doneWaiting(std::__exception_ptr::exception_ptr)::{lambda()#1}>::execute(tbb::detail::d1::execution_data&) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week1/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_MULTIARCHS_X_2024-08-18-0000/lib/el8_amd64_gcc12/scram_x86-64-v3/libFWCoreFramework.so
#6  0x0000153a33d103e1 in tbb::detail::r1::task_dispatcher::local_wait_for_all<false, tbb::detail::r1::external_waiter> (waiter=..., t=<optimized out>, this=0x153a307bb900) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-2391c941213c757dc9a1835b31681235/tbb-v2021.9.0/src/tbb/task_dispatcher.h:322
#7  tbb::detail::r1::task_dispatcher::local_wait_for_all<tbb::detail::r1::external_waiter> (waiter=..., t=<optimized out>, this=0x153a307bb900) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-2391c941213c757dc9a1835b31681235/tbb-v2021.9.0/src/tbb/task_dispatcher.h:458
#8  tbb::detail::r1::task_dispatcher::execute_and_wait (t=<optimized out>, wait_ctx=..., w_ctx=...) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-2391c941213c757dc9a1835b31681235/tbb-v2021.9.0/src/tbb/task_dispatcher.cpp:168
#9  0x0000153a34948f7b in edm::FinalWaitingTask::wait() () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week1/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_MULTIARCHS_X_2024-08-18-0000/lib/el8_amd64_gcc12/scram_x86-64-v3/libFWCoreFramework.so
#10 0x0000153a34955f96 in edm::EventProcessor::endUnfinishedRun(bool) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week1/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_MULTIARCHS_X_2024-08-18-0000/lib/el8_amd64_gcc12/scram_x86-64-v3/libFWCoreFramework.so
#11 0x0000153a3495734d in edm::EventProcessor::runToCompletion() () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week1/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_MULTIARCHS_X_2024-08-18-0000/lib/el8_amd64_gcc12/scram_x86-64-v3/libFWCoreFramework.so
#12 0x000000000040840c in tbb::detail::d1::task_arena_function<main::{lambda()#1}::operator()() const::{lambda()#1}, void>::operator()() const ()
#13 0x0000153a33cfc9ad in tbb::detail::r1::task_arena_impl::execute (ta=..., d=...) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-2391c941213c757dc9a1835b31681235/tbb-v2021.9.0/src/tbb/arena.cpp:688
#14 0x000000000040a0f2 in main::{lambda()#1}::operator()() const ()
#15 0x0000000000405100 in main ()

Current Modules:

Module: none (crashed)
Module: none
Module: none
Module: none
Module: none
Module: none
Module: none
Module: none

Doesn't reproduce when ran locally

@smuzaffar
Copy link
Contributor

#42093 (comment) was run on a cmsbuild926 (16 cores AlmaLinux9 node). Any idea why it is using 26 threads? I see that when this tests was running/failed then system log contains message [a]. Note that scram was running 16 unit tests in parallel and then if some unit tests starting using multiple theads then that can overload system and can hit system limits.

Is there any way to configure this unit test to not run multi-threaded ?

[a]

Aug 18 13:15:38 cmsbuild926.cern.ch kernel: cmsRun[3435603]: segfault at 10 ip 0000153a33d0de7d sp 00001539db5f5dd0 error 6 in libtbb.so.12.9[153a33cf3000+1f000] likely on CPU 7 (core 0, socket 7)
Aug 18 13:15:38 cmsbuild926.cern.ch kernel: Code: 19 01 00 e8 b5 56 fe ff 48 89 c3 48 85 c0 74 3d 48 89 de 4c 89 e7 e8 82 eb ff ff 48 8b 43 18 4c 8b 6b 20 48 89 ee 48 8b 7b 28 <4c> 89 65 10 48 8b 40 20 48 89 45 20 e8 32 e1 ff ff 48 83 c4 08 4c
Aug 18 13:15:38 cmsbuild926.cern.ch systemd[1]: Started Process Core Dump (PID 3436434/UID 0).
Aug 18 13:15:38 cmsbuild926.cern.ch systemd-coredump[3436435]: Resource limits disable core dumping for process 3434931 (cmsRun).
Aug 18 13:15:38 cmsbuild926.cern.ch systemd-coredump[3436435]: Process 3434931 (cmsRun) of user 501 dumped core.

@Dr15Jones
Copy link
Contributor

Looking at the traceback, the threads are

  • 3 for Eigen. We have never found a way to stop those but I think they never do any work.
  • 14 xrootd threads. I do not know why there are so many but the documentation about xrootd client seems to imply the number can be controlled by an environment variable
  • 1 watcher thread used to do stack traces if the job crashes
  • 8 TBB threads. This can be controlled via the job configuration.

@iarspider
Copy link
Contributor

@iarspider
Copy link
Contributor

On an unrelated note: the DBG_X log file for DQM/Integration unit tests is over 11M lines long (compared to 26k+ lines for non-DBG builds), seems a bit too verbose to me.

@nothingface0
Copy link
Contributor

On an unrelated note: the DBG_X log file for DQM/Integration unit tests is over 11M lines long (compared to 26k+ lines for non-DBG builds), seems a bit too verbose to me.

I saw too many ChanStatusForCaloTower in those logs you linked (probably printed from here).

@iarspider
Copy link
Contributor

@makortel
Copy link
Contributor Author

makortel commented Nov 13, 2024

Crashes in #46685 (comment)

Thread 10 (Thread 0x14fba0ffe700 (LWP 433408) "cmsRun"):
#0  0x000014fbf8190098 in nanosleep () from /lib64/libc.so.6
#1  0x000014fbf818ff9e in sleep () from /lib64/libc.so.6
#2  0x000014fbf2aba1d0 in sig_pause_for_stacktrace () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02863/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-11-10-0000/lib/el8_amd64_gcc12/pluginFWCoreServicesPlugins.so
#3  <signal handler called>
#4  0x000014fbfb1d4a8d in tbb::detail::d1::function_task<edm::SerialTaskQueue::spawn(edm::SerialTaskQueue::TaskBase&)::{lambda()#1}>::execute(tbb::detail::d1::execution_data&) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02863/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-11-10-0000/lib/el8_amd64_gcc12/libFWCoreConcurrency.so
#5  0x000014fbfa924b3b in tbb::detail::r1::task_dispatcher::local_wait_for_all<false, tbb::detail::r1::outermost_worker_waiter> (t=0x14fb5dace200, waiter=..., this=0x14fbf6bbbc80) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-2391c941213c757dc9a1835b31681235/tbb-v2021.9.0/src/tbb/task_dispatcher.h:322
#6  tbb::detail::r1::task_dispatcher::local_wait_for_all<tbb::detail::r1::outermost_worker_waiter> (t=0x0, waiter=..., this=0x14fbf6bbbc80) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-2391c941213c757dc9a1835b31681235/tbb-v2021.9.0/src/tbb/task_dispatcher.h:458
#7  tbb::detail::r1::arena::process (tls=..., this=<optimized out>) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-2391c941213c757dc9a1835b31681235/tbb-v2021.9.0/src/tbb/arena.cpp:137
#8  tbb::detail::r1::market::process (this=<optimized out>, j=...) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-2391c941213c757dc9a1835b31681235/tbb-v2021.9.0/src/tbb/market.cpp:599
#9  0x000014fbfa926cee in tbb::detail::r1::rml::private_worker::run (this=0x14fbf2face00) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-2391c941213c757dc9a1835b31681235/tbb-v2021.9.0/src/tbb/private_server.cpp:271
#10 tbb::detail::r1::rml::private_worker::thread_routine (arg=0x14fbf2face00) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-2391c941213c757dc9a1835b31681235/tbb-v2021.9.0/src/tbb/private_server.cpp:221
#11 0x000014fbf84661ca in start_thread () from /lib64/libpthread.so.0
#12 0x000014fbf80c18d3 in clone () from /lib64/libc.so.6

Thread 9 (Thread 0x14fba13ff700 (LWP 433407) "cmsRun"):
#0  0x000014fbf81c7307 in epoll_wait () from /lib64/libc.so.6
#1  0x000014fbf264ab62 in XrdSys::IOEvents::PollE::Begin(XrdSysSemaphore*, int&, char const**) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02863/el8_amd64_gcc12/cms/cmssw-patch/CMSSW_14_1_X_2024-11-13-1100/external/el8_amd64_gcc12/lib/libXrdUtils.so.3
#2  0x000014fbf2646acd in XrdSys::IOEvents::BootStrap::Start(void*) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02863/el8_amd64_gcc12/cms/cmssw-patch/CMSSW_14_1_X_2024-11-13-1100/external/el8_amd64_gcc12/lib/libXrdUtils.so.3
#3  0x000014fbf264fef7 in XrdSysThread_Xeq () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02863/el8_amd64_gcc12/cms/cmssw-patch/CMSSW_14_1_X_2024-11-13-1100/external/el8_amd64_gcc12/lib/libXrdUtils.so.3
#4  0x000014fbf84661ca in start_thread () from /lib64/libpthread.so.0
#5  0x000014fbf80c18d3 in clone () from /lib64/libc.so.6

Thread 8 (Thread 0x14fba21fe700 (LWP 433406) "cmsRun"):
#0  0x000014fbf8190098 in nanosleep () from /lib64/libc.so.6
#1  0x000014fbf818ff9e in sleep () from /lib64/libc.so.6
#2  0x000014fbf2aba1d0 in sig_pause_for_stacktrace () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02863/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-11-10-0000/lib/el8_amd64_gcc12/pluginFWCoreServicesPlugins.so
#3  <signal handler called>
#4  0x000014fbfaf5ce4d in std::__weak_ptr<edm::serviceregistry::ServicesManager, (__gnu_cxx::_Lock_policy)2>::__weak_ptr(std::__weak_ptr<edm::serviceregistry::ServicesManager, (__gnu_cxx::_Lock_policy)2> const&) [clone .lto_priv.0] () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02863/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-11-10-0000/lib/el8_amd64_gcc12/libFWCoreFramework.so
#5  0x000014fbfaf5f8f6 in edm::StreamSchedule::processOneStreamAsync<edm::OccurrenceTraits<edm::RunPrincipal, (edm::BranchActionType)2> >(edm::WaitingTaskHolder, edm::OccurrenceTraits<edm::RunPrincipal, (edm::BranchActionType)2>::TransitionInfoType&, edm::ServiceToken const&, bool)::{lambda()#2}::operator()() () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02863/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-11-10-0000/lib/el8_amd64_gcc12/libFWCoreFramework.so
#6  0x000014fbfaf6ed37 in tbb::detail::d1::enqueue_task<edm::StreamSchedule::processOneStreamAsync<edm::OccurrenceTraits<edm::RunPrincipal, (edm::BranchActionType)2> >(edm::WaitingTaskHolder, edm::OccurrenceTraits<edm::RunPrincipal, (edm::BranchActionType)2>::TransitionInfoType&, edm::ServiceToken const&, bool)::{lambda()#4}>::execute(tbb::detail::d1::execution_data&) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02863/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-11-10-0000/lib/el8_amd64_gcc12/libFWCoreFramework.so
#7  0x000014fbfa924b3b in tbb::detail::r1::task_dispatcher::local_wait_for_all<false, tbb::detail::r1::outermost_worker_waiter> (t=0x14fb5dace700, waiter=..., this=0x14fbf6bbbb80) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-2391c941213c757dc9a1835b31681235/tbb-v2021.9.0/src/tbb/task_dispatcher.h:322
#8  tbb::detail::r1::task_dispatcher::local_wait_for_all<tbb::detail::r1::outermost_worker_waiter> (t=0x0, waiter=..., this=0x14fbf6bbbb80) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-2391c941213c757dc9a1835b31681235/tbb-v2021.9.0/src/tbb/task_dispatcher.h:458
#9  tbb::detail::r1::arena::process (tls=..., this=<optimized out>) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-2391c941213c757dc9a1835b31681235/tbb-v2021.9.0/src/tbb/arena.cpp:137
#10 tbb::detail::r1::market::process (this=<optimized out>, j=...) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-2391c941213c757dc9a1835b31681235/tbb-v2021.9.0/src/tbb/market.cpp:599
#11 0x000014fbfa926cee in tbb::detail::r1::rml::private_worker::run (this=0x14fbf2facf00) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-2391c941213c757dc9a1835b31681235/tbb-v2021.9.0/src/tbb/private_server.cpp:271
#12 tbb::detail::r1::rml::private_worker::thread_routine (arg=0x14fbf2facf00) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-2391c941213c757dc9a1835b31681235/tbb-v2021.9.0/src/tbb/private_server.cpp:221
#13 0x000014fbf84661ca in start_thread () from /lib64/libpthread.so.0
#14 0x000014fbf80c18d3 in clone () from /lib64/libc.so.6

Thread 7 (Thread 0x14fba2bff700 (LWP 433405) "cmsRun"):
#0  0x000014fbf8190098 in nanosleep () from /lib64/libc.so.6
#1  0x000014fbf818ff9e in sleep () from /lib64/libc.so.6
#2  0x000014fbf2aba1d0 in sig_pause_for_stacktrace () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02863/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-11-10-0000/lib/el8_amd64_gcc12/pluginFWCoreServicesPlugins.so
#3  <signal handler called>
#4  0x000014fbfb04054d in edm::WorkerT<edm::stream::EDProducerAdaptorBase>::implDoStreamEnd(edm::StreamID, edm::RunTransitionInfo const&, edm::ModuleCallingContext const*) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02863/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-11-10-0000/lib/el8_amd64_gcc12/libFWCoreFramework.so
#5  0x000014fbfaf626a0 in edm::Worker::doWorkNoPrefetchingAsync<edm::OccurrenceTraits<edm::RunPrincipal, (edm::BranchActionType)2> >(edm::WaitingTaskHolder, edm::OccurrenceTraits<edm::RunPrincipal, (edm::BranchActionType)2>::TransitionInfoType const&, edm::ServiceToken const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::RunPrincipal, (edm::BranchActionType)2>::Context const*)::{lambda()#1}::operator()() const () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02863/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-11-10-0000/lib/el8_amd64_gcc12/libFWCoreFramework.so
#6  0x000014fbfaf6e8e9 in tbb::detail::d1::function_task<edm::Worker::doWorkNoPrefetchingAsync<edm::OccurrenceTraits<edm::RunPrincipal, (edm::BranchActionType)2> >(edm::WaitingTaskHolder, edm::OccurrenceTraits<edm::RunPrincipal, (edm::BranchActionType)2>::TransitionInfoType const&, edm::ServiceToken const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::RunPrincipal, (edm::BranchActionType)2>::Context const*)::{lambda()#1}>::execute(tbb::detail::d1::execution_data&) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02863/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-11-10-0000/lib/el8_amd64_gcc12/libFWCoreFramework.so
#7  0x000014fbfa924b3b in tbb::detail::r1::task_dispatcher::local_wait_for_all<false, tbb::detail::r1::outermost_worker_waiter> (t=0x14fb5da15a00, waiter=..., this=0x14fbf6bbba80) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-2391c941213c757dc9a1835b31681235/tbb-v2021.9.0/src/tbb/task_dispatcher.h:322
#8  tbb::detail::r1::task_dispatcher::local_wait_for_all<tbb::detail::r1::outermost_worker_waiter> (t=0x0, waiter=..., this=0x14fbf6bbba80) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-2391c941213c757dc9a1835b31681235/tbb-v2021.9.0/src/tbb/task_dispatcher.h:458
#9  tbb::detail::r1::arena::process (tls=..., this=<optimized out>) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-2391c941213c757dc9a1835b31681235/tbb-v2021.9.0/src/tbb/arena.cpp:137
#10 tbb::detail::r1::market::process (this=<optimized out>, j=...) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-2391c941213c757dc9a1835b31681235/tbb-v2021.9.0/src/tbb/market.cpp:599
#11 0x000014fbfa926cee in tbb::detail::r1::rml::private_worker::run (this=0x14fbf2face80) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-2391c941213c757dc9a1835b31681235/tbb-v2021.9.0/src/tbb/private_server.cpp:271
#12 tbb::detail::r1::rml::private_worker::thread_routine (arg=0x14fbf2face80) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-2391c941213c757dc9a1835b31681235/tbb-v2021.9.0/src/tbb/private_server.cpp:221
#13 0x000014fbf84661ca in start_thread () from /lib64/libpthread.so.0
#14 0x000014fbf80c18d3 in clone () from /lib64/libc.so.6

Thread 6 (Thread 0x14fba3bfe700 (LWP 433404) "cmsRun"):
#0  0x000014fbf81baac1 in poll () from /lib64/libc.so.6
#1  0x000014fbf2abd857 in edm::service::InitRootHandlers::stacktraceFromThread() () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02863/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-11-10-0000/lib/el8_amd64_gcc12/pluginFWCoreServicesPlugins.so
#2  0x000014fbf2abda54 in sig_dostack_then_abort () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02863/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-11-10-0000/lib/el8_amd64_gcc12/pluginFWCoreServicesPlugins.so
#3  <signal handler called>
#4  tbb::detail::r1::spawn (t=..., ctx=...) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-2391c941213c757dc9a1835b31681235/tbb-v2021.9.0/src/tbb/task_dispatcher.cpp:36
#5  0x000014fbfaf2e608 in tbb::detail::d1::function_task<edm::WaitingTaskHolder::doneWaiting(std::__exception_ptr::exception_ptr)::{lambda()#1}>::execute(tbb::detail::d1::execution_data&) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02863/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-11-10-0000/lib/el8_amd64_gcc12/libFWCoreFramework.so
#6  0x000014fbfa924b3b in tbb::detail::r1::task_dispatcher::local_wait_for_all<false, tbb::detail::r1::outermost_worker_waiter> (t=0x14fb5dabb000, waiter=..., this=0x14fbf6bbba00) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-2391c941213c757dc9a1835b31681235/tbb-v2021.9.0/src/tbb/task_dispatcher.h:322
#7  tbb::detail::r1::task_dispatcher::local_wait_for_all<tbb::detail::r1::outermost_worker_waiter> (t=0x0, waiter=..., this=0x14fbf6bbba00) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-2391c941213c757dc9a1835b31681235/tbb-v2021.9.0/src/tbb/task_dispatcher.h:458
#8  tbb::detail::r1::arena::process (tls=..., this=<optimized out>) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-2391c941213c757dc9a1835b31681235/tbb-v2021.9.0/src/tbb/arena.cpp:137
#9  tbb::detail::r1::market::process (this=<optimized out>, j=...) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-2391c941213c757dc9a1835b31681235/tbb-v2021.9.0/src/tbb/market.cpp:599
#10 0x000014fbfa926cee in tbb::detail::r1::rml::private_worker::run (this=0x14fbf2fad000) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-2391c941213c757dc9a1835b31681235/tbb-v2021.9.0/src/tbb/private_server.cpp:271
#11 tbb::detail::r1::rml::private_worker::thread_routine (arg=0x14fbf2fad000) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-2391c941213c757dc9a1835b31681235/tbb-v2021.9.0/src/tbb/private_server.cpp:221
#12 0x000014fbf84661ca in start_thread () from /lib64/libpthread.so.0
#13 0x000014fbf80c18d3 in clone () from /lib64/libc.so.6

Thread 5 (Thread 0x14fba45ff700 (LWP 433403) "cmsRun"):
#0  0x000014fbf8190098 in nanosleep () from /lib64/libc.so.6
#1  0x000014fbf818ff9e in sleep () from /lib64/libc.so.6
#2  0x000014fbf2aba1d0 in sig_pause_for_stacktrace () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02863/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-11-10-0000/lib/el8_amd64_gcc12/pluginFWCoreServicesPlugins.so
#3  <signal handler called>
#4  0x000014fbfb37997e in std::_Sp_counted_base<(__gnu_cxx::_Lock_policy)2>::_M_release() () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02863/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-11-10-0000/lib/el8_amd64_gcc12/libFWCoreServiceRegistry.so
#5  0x000014fbfb379afa in edm::ServiceRegistry::unsetContext(edm::ServiceToken const&) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02863/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-11-10-0000/lib/el8_amd64_gcc12/libFWCoreServiceRegistry.so
#6  0x000014fbfaf443e4 in edm::ServiceRegistry::Operate::~Operate() [clone .lto_priv.0] () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02863/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-11-10-0000/lib/el8_amd64_gcc12/libFWCoreFramework.so
#7  0x000014fbfaf62607 in edm::Worker::doWorkNoPrefetchingAsync<edm::OccurrenceTraits<edm::RunPrincipal, (edm::BranchActionType)2> >(edm::WaitingTaskHolder, edm::OccurrenceTraits<edm::RunPrincipal, (edm::BranchActionType)2>::TransitionInfoType const&, edm::ServiceToken const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::RunPrincipal, (edm::BranchActionType)2>::Context const*)::{lambda()#1}::operator()() const () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02863/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-11-10-0000/lib/el8_amd64_gcc12/libFWCoreFramework.so
#8  0x000014fbfaf6e8e9 in tbb::detail::d1::function_task<edm::Worker::doWorkNoPrefetchingAsync<edm::OccurrenceTraits<edm::RunPrincipal, (edm::BranchActionType)2> >(edm::WaitingTaskHolder, edm::OccurrenceTraits<edm::RunPrincipal, (edm::BranchActionType)2>::TransitionInfoType const&, edm::ServiceToken const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::RunPrincipal, (edm::BranchActionType)2>::Context const*)::{lambda()#1}>::execute(tbb::detail::d1::execution_data&) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02863/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-11-10-0000/lib/el8_amd64_gcc12/libFWCoreFramework.so
#9  0x000014fbfa924b3b in tbb::detail::r1::task_dispatcher::local_wait_for_all<false, tbb::detail::r1::outermost_worker_waiter> (t=0x14fb5db23900, waiter=..., this=0x14fbf6bbbc00) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-2391c941213c757dc9a1835b31681235/tbb-v2021.9.0/src/tbb/task_dispatcher.h:322
#10 tbb::detail::r1::task_dispatcher::local_wait_for_all<tbb::detail::r1::outermost_worker_waiter> (t=0x0, waiter=..., this=0x14fbf6bbbc00) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-2391c941213c757dc9a1835b31681235/tbb-v2021.9.0/src/tbb/task_dispatcher.h:458
#11 tbb::detail::r1::arena::process (tls=..., this=<optimized out>) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-2391c941213c757dc9a1835b31681235/tbb-v2021.9.0/src/tbb/arena.cpp:137
#12 tbb::detail::r1::market::process (this=<optimized out>, j=...) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-2391c941213c757dc9a1835b31681235/tbb-v2021.9.0/src/tbb/market.cpp:599
#13 0x000014fbfa926cee in tbb::detail::r1::rml::private_worker::run (this=0x14fbf2facf80) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-2391c941213c757dc9a1835b31681235/tbb-v2021.9.0/src/tbb/private_server.cpp:271
#14 tbb::detail::r1::rml::private_worker::thread_routine (arg=0x14fbf2facf80) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-2391c941213c757dc9a1835b31681235/tbb-v2021.9.0/src/tbb/private_server.cpp:221
#15 0x000014fbf84661ca in start_thread () from /lib64/libpthread.so.0
#16 0x000014fbf80c18d3 in clone () from /lib64/libc.so.6

Thread 4 (Thread 0x14fba559f700 (LWP 433402) "cmsRun"):
#0  0x000014fbf8190098 in nanosleep () from /lib64/libc.so.6
#1  0x000014fbf818ff9e in sleep () from /lib64/libc.so.6
#2  0x000014fbf2aba1d0 in sig_pause_for_stacktrace () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02863/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-11-10-0000/lib/el8_amd64_gcc12/pluginFWCoreServicesPlugins.so
#3  <signal handler called>
#4  0x000014fbfb1d5fd5 in edm::WaitingTaskList::announce() () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02863/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-11-10-0000/lib/el8_amd64_gcc12/libFWCoreConcurrency.so
#5  0x000014fbfaf62630 in edm::Worker::doWorkNoPrefetchingAsync<edm::OccurrenceTraits<edm::RunPrincipal, (edm::BranchActionType)2> >(edm::WaitingTaskHolder, edm::OccurrenceTraits<edm::RunPrincipal, (edm::BranchActionType)2>::TransitionInfoType const&, edm::ServiceToken const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::RunPrincipal, (edm::BranchActionType)2>::Context const*)::{lambda()#1}::operator()() const () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02863/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-11-10-0000/lib/el8_amd64_gcc12/libFWCoreFramework.so
#6  0x000014fbfaf6e8e9 in tbb::detail::d1::function_task<edm::Worker::doWorkNoPrefetchingAsync<edm::OccurrenceTraits<edm::RunPrincipal, (edm::BranchActionType)2> >(edm::WaitingTaskHolder, edm::OccurrenceTraits<edm::RunPrincipal, (edm::BranchActionType)2>::TransitionInfoType const&, edm::ServiceToken const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::RunPrincipal, (edm::BranchActionType)2>::Context const*)::{lambda()#1}>::execute(tbb::detail::d1::execution_data&) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02863/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-11-10-0000/lib/el8_amd64_gcc12/libFWCoreFramework.so
#7  0x000014fbfa924b3b in tbb::detail::r1::task_dispatcher::local_wait_for_all<false, tbb::detail::r1::outermost_worker_waiter> (t=0x14fb5d9b7500, waiter=..., this=0x14fbf6bbb980) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-2391c941213c757dc9a1835b31681235/tbb-v2021.9.0/src/tbb/task_dispatcher.h:322
#8  tbb::detail::r1::task_dispatcher::local_wait_for_all<tbb::detail::r1::outermost_worker_waiter> (t=0x0, waiter=..., this=0x14fbf6bbb980) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-2391c941213c757dc9a1835b31681235/tbb-v2021.9.0/src/tbb/task_dispatcher.h:458
#9  tbb::detail::r1::arena::process (tls=..., this=<optimized out>) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-2391c941213c757dc9a1835b31681235/tbb-v2021.9.0/src/tbb/arena.cpp:137
#10 tbb::detail::r1::market::process (this=<optimized out>, j=...) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-2391c941213c757dc9a1835b31681235/tbb-v2021.9.0/src/tbb/market.cpp:599
#11 0x000014fbfa926cee in tbb::detail::r1::rml::private_worker::run (this=0x14fbf2fad100) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-2391c941213c757dc9a1835b31681235/tbb-v2021.9.0/src/tbb/private_server.cpp:271
#12 tbb::detail::r1::rml::private_worker::thread_routine (arg=0x14fbf2fad100) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-2391c941213c757dc9a1835b31681235/tbb-v2021.9.0/src/tbb/private_server.cpp:221
#13 0x000014fbf84661ca in start_thread () from /lib64/libpthread.so.0
#14 0x000014fbf80c18d3 in clone () from /lib64/libc.so.6

Thread 3 (Thread 0x14fba5fa0700 (LWP 433401) "cmsRun"):
#0  0x000014fbf8190098 in nanosleep () from /lib64/libc.so.6
#1  0x000014fbf818ff9e in sleep () from /lib64/libc.so.6
#2  0x000014fbf2aba1d0 in sig_pause_for_stacktrace () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02863/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-11-10-0000/lib/el8_amd64_gcc12/pluginFWCoreServicesPlugins.so
#3  <signal handler called>
#4  0x000014fbfaf54bca in std::_Sp_counted_base<(__gnu_cxx::_Lock_policy)2>::_M_weak_release() () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02863/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-11-10-0000/lib/el8_amd64_gcc12/libFWCoreFramework.so
#5  0x000014fbfaf5f053 in edm::StreamSchedule::processOneStreamAsync<edm::OccurrenceTraits<edm::RunPrincipal, (edm::BranchActionType)2> >(edm::WaitingTaskHolder, edm::OccurrenceTraits<edm::RunPrincipal, (edm::BranchActionType)2>::TransitionInfoType&, edm::ServiceToken const&, bool)::{lambda()#2}::operator()() () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02863/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-11-10-0000/lib/el8_amd64_gcc12/libFWCoreFramework.so
#6  0x000014fbfaf6ed37 in tbb::detail::d1::enqueue_task<edm::StreamSchedule::processOneStreamAsync<edm::OccurrenceTraits<edm::RunPrincipal, (edm::BranchActionType)2> >(edm::WaitingTaskHolder, edm::OccurrenceTraits<edm::RunPrincipal, (edm::BranchActionType)2>::TransitionInfoType&, edm::ServiceToken const&, bool)::{lambda()#4}>::execute(tbb::detail::d1::execution_data&) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02863/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-11-10-0000/lib/el8_amd64_gcc12/libFWCoreFramework.so
#7  0x000014fbfa924b3b in tbb::detail::r1::task_dispatcher::local_wait_for_all<false, tbb::detail::r1::outermost_worker_waiter> (t=0x14fb5daea900, waiter=..., this=0x14fbf6bbbb00) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-2391c941213c757dc9a1835b31681235/tbb-v2021.9.0/src/tbb/task_dispatcher.h:322
#8  tbb::detail::r1::task_dispatcher::local_wait_for_all<tbb::detail::r1::outermost_worker_waiter> (t=0x0, waiter=..., this=0x14fbf6bbbb00) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-2391c941213c757dc9a1835b31681235/tbb-v2021.9.0/src/tbb/task_dispatcher.h:458
#9  tbb::detail::r1::arena::process (tls=..., this=<optimized out>) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-2391c941213c757dc9a1835b31681235/tbb-v2021.9.0/src/tbb/arena.cpp:137
#10 tbb::detail::r1::market::process (this=<optimized out>, j=...) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-2391c941213c757dc9a1835b31681235/tbb-v2021.9.0/src/tbb/market.cpp:599
#11 0x000014fbfa926cee in tbb::detail::r1::rml::private_worker::run (this=0x14fbf2fad080) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-2391c941213c757dc9a1835b31681235/tbb-v2021.9.0/src/tbb/private_server.cpp:271
#12 tbb::detail::r1::rml::private_worker::thread_routine (arg=0x14fbf2fad080) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-2391c941213c757dc9a1835b31681235/tbb-v2021.9.0/src/tbb/private_server.cpp:221
#13 0x000014fbf84661ca in start_thread () from /lib64/libpthread.so.0
#14 0x000014fbf80c18d3 in clone () from /lib64/libc.so.6

Thread 2 (Thread 0x14fbceba9700 (LWP 433308) "cmsRun"):
#0  0x000014fbf84706a2 in waitpid () from /lib64/libpthread.so.0
#1  0x000014fbf2aba327 in edm::service::cmssw_stacktrace_fork() () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02863/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-11-10-0000/lib/el8_amd64_gcc12/pluginFWCoreServicesPlugins.so
#2  0x000014fbf2abd63a in edm::service::InitRootHandlers::stacktraceHelperThread() () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02863/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-11-10-0000/lib/el8_amd64_gcc12/pluginFWCoreServicesPlugins.so
#3  0x000014fbf8ad8a73 in std::execute_native_thread_routine (__p=0x14fbd2196ea0) at ../../../../../libstdc++-v3/src/c++11/thread.cc:82
#4  0x000014fbf84661ca in start_thread () from /lib64/libpthread.so.0
#5  0x000014fbf80c18d3 in clone () from /lib64/libc.so.6

Thread 1 (Thread 0x14fbfa0f7680 (LWP 432190) "cmsRun"):
#0  0x000014fbf8190098 in nanosleep () from /lib64/libc.so.6
#1  0x000014fbf818ff9e in sleep () from /lib64/libc.so.6
#2  0x000014fbf2aba1d0 in sig_pause_for_stacktrace () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02863/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-11-10-0000/lib/el8_amd64_gcc12/pluginFWCoreServicesPlugins.so
#3  <signal handler called>
#4  0x000014fbf846da65 in pthread_kill () from /lib64/libpthread.so.0
#5  0x000014fbf2abde88 in sig_dostack_then_abort () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02863/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-11-10-0000/lib/el8_amd64_gcc12/pluginFWCoreServicesPlugins.so
#6  <signal handler called>
#7  tbb::detail::r1::spawn (t=..., ctx=...) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-2391c941213c757dc9a1835b31681235/tbb-v2021.9.0/src/tbb/task_dispatcher.cpp:36
#8  0x000014fbfaf2e608 in tbb::detail::d1::function_task<edm::WaitingTaskHolder::doneWaiting(std::__exception_ptr::exception_ptr)::{lambda()#1}>::execute(tbb::detail::d1::execution_data&) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02863/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-11-10-0000/lib/el8_amd64_gcc12/libFWCoreFramework.so
#9  0x000014fbfa92d3e1 in tbb::detail::r1::task_dispatcher::local_wait_for_all<false, tbb::detail::r1::external_waiter> (waiter=..., t=<optimized out>, this=0x14fbf6bbb900) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-2391c941213c757dc9a1835b31681235/tbb-v2021.9.0/src/tbb/task_dispatcher.h:322
#10 tbb::detail::r1::task_dispatcher::local_wait_for_all<tbb::detail::r1::external_waiter> (waiter=..., t=<optimized out>, this=0x14fbf6bbb900) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-2391c941213c757dc9a1835b31681235/tbb-v2021.9.0/src/tbb/task_dispatcher.h:458
#11 tbb::detail::r1::task_dispatcher::execute_and_wait (t=<optimized out>, wait_ctx=..., w_ctx=...) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-2391c941213c757dc9a1835b31681235/tbb-v2021.9.0/src/tbb/task_dispatcher.cpp:168
#12 0x000014fbfaf4450b in edm::FinalWaitingTask::wait() () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02863/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-11-10-0000/lib/el8_amd64_gcc12/libFWCoreFramework.so
#13 0x000014fbfaf51196 in edm::EventProcessor::endUnfinishedRun(bool) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02863/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-11-10-0000/lib/el8_amd64_gcc12/libFWCoreFramework.so
#14 0x000014fbfaf524ed in edm::EventProcessor::runToCompletion() () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02863/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-11-10-0000/lib/el8_amd64_gcc12/libFWCoreFramework.so
#15 0x000000000040840c in tbb::detail::d1::task_arena_function<main::{lambda()#1}::operator()() const::{lambda()#1}, void>::operator()() const ()
#16 0x000014fbfa9199ad in tbb::detail::r1::task_arena_impl::execute (ta=..., d=...) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-2391c941213c757dc9a1835b31681235/tbb-v2021.9.0/src/tbb/arena.cpp:688
#17 0x000000000040a0f2 in main::{lambda()#1}::operator()() const ()
#18 0x0000000000405100 in main ()

Current Modules:

Module: none (crashed)

sig_dostack_then_abort is shown in threads 6 and 1


Thread 10 (Thread 0x14fba0ffe700 (LWP 433408) "cmsRun"):
#0  0x000014fbf8190098 in nanosleep () from /lib64/libc.so.6
#1  0x000014fbf818ff9e in sleep () from /lib64/libc.so.6
#2  0x000014fbf2aba1d0 in sig_pause_for_stacktrace () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02863/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-11-10-0000/lib/el8_amd64_gcc12/pluginFWCoreServicesPlugins.so
#3  <signal handler called>
#4  0x000014fbfb378a20 in edm::ESParentContext::ESParentContext(edm::ModuleCallingContext const*) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02863/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-11-10-0000/lib/el8_amd64_gcc12/libFWCoreServiceRegistry.so
#5  0x000014fbfb040575 in edm::WorkerT<edm::stream::EDProducerAdaptorBase>::implDoStreamEnd(edm::StreamID, edm::RunTransitionInfo const&, edm::ModuleCallingContext const*) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02863/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-11-10-0000/lib/el8_amd64_gcc12/libFWCoreFramework.so
#6  0x000014fbfaf626a0 in edm::Worker::doWorkNoPrefetchingAsync<edm::OccurrenceTraits<edm::RunPrincipal, (edm::BranchActionType)2> >(edm::WaitingTaskHolder, edm::OccurrenceTraits<edm::RunPrincipal, (edm::BranchActionType)2>::TransitionInfoType const&, edm::ServiceToken const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::RunPrincipal, (edm::BranchActionType)2>::Context const*)::{lambda()#1}::operator()() const () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02863/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-11-10-0000/lib/el8_amd64_gcc12/libFWCoreFramework.so
#7  0x000014fbfaf6e8e9 in tbb::detail::d1::function_task<edm::Worker::doWorkNoPrefetchingAsync<edm::OccurrenceTraits<edm::RunPrincipal, (edm::BranchActionType)2> >(edm::WaitingTaskHolder, edm::OccurrenceTraits<edm::RunPrincipal, (edm::BranchActionType)2>::TransitionInfoType const&, edm::ServiceToken const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::RunPrincipal, (edm::BranchActionType)2>::Context const*)::{lambda()#1}>::execute(tbb::detail::d1::execution_data&) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02863/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-11-10-0000/lib/el8_amd64_gcc12/libFWCoreFramework.so
#8  0x000014fbfa924b3b in tbb::detail::r1::task_dispatcher::local_wait_for_all<false, tbb::detail::r1::outermost_worker_waiter> (t=0x14fb5d9b7000, waiter=..., this=0x14fbf6bbbc80) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-2391c941213c757dc9a1835b31681235/tbb-v2021.9.0/src/tbb/task_dispatcher.h:322
#9  tbb::detail::r1::task_dispatcher::local_wait_for_all<tbb::detail::r1::outermost_worker_waiter> (t=0x0, waiter=..., this=0x14fbf6bbbc80) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-2391c941213c757dc9a1835b31681235/tbb-v2021.9.0/src/tbb/task_dispatcher.h:458
#10 tbb::detail::r1::arena::process (tls=..., this=<optimized out>) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-2391c941213c757dc9a1835b31681235/tbb-v2021.9.0/src/tbb/arena.cpp:137
#11 tbb::detail::r1::market::process (this=<optimized out>, j=...) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-2391c941213c757dc9a1835b31681235/tbb-v2021.9.0/src/tbb/market.cpp:599
#12 0x000014fbfa926cee in tbb::detail::r1::rml::private_worker::run (this=0x14fbf2face00) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-2391c941213c757dc9a1835b31681235/tbb-v2021.9.0/src/tbb/private_server.cpp:271
#13 tbb::detail::r1::rml::private_worker::thread_routine (arg=0x14fbf2face00) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-2391c941213c757dc9a1835b31681235/tbb-v2021.9.0/src/tbb/private_server.cpp:221
#14 0x000014fbf84661ca in start_thread () from /lib64/libpthread.so.0
#15 0x000014fbf80c18d3 in clone () from /lib64/libc.so.6

Thread 9 (Thread 0x14fba13ff700 (LWP 433407) "cmsRun"):
#0  0x000014fbf81c7307 in epoll_wait () from /lib64/libc.so.6
#1  0x000014fbf264ab62 in XrdSys::IOEvents::PollE::Begin(XrdSysSemaphore*, int&, char const**) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02863/el8_amd64_gcc12/cms/cmssw-patch/CMSSW_14_1_X_2024-11-13-1100/external/el8_amd64_gcc12/lib/libXrdUtils.so.3
#2  0x000014fbf2646acd in XrdSys::IOEvents::BootStrap::Start(void*) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02863/el8_amd64_gcc12/cms/cmssw-patch/CMSSW_14_1_X_2024-11-13-1100/external/el8_amd64_gcc12/lib/libXrdUtils.so.3
#3  0x000014fbf264fef7 in XrdSysThread_Xeq () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02863/el8_amd64_gcc12/cms/cmssw-patch/CMSSW_14_1_X_2024-11-13-1100/external/el8_amd64_gcc12/lib/libXrdUtils.so.3
#4  0x000014fbf84661ca in start_thread () from /lib64/libpthread.so.0
#5  0x000014fbf80c18d3 in clone () from /lib64/libc.so.6

Thread 8 (Thread 0x14fba21fe700 (LWP 433406) "cmsRun"):
#0  0x000014fbf8190098 in nanosleep () from /lib64/libc.so.6
#1  0x000014fbf818ff9e in sleep () from /lib64/libc.so.6
#2  0x000014fbf2aba1d0 in sig_pause_for_stacktrace () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02863/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-11-10-0000/lib/el8_amd64_gcc12/pluginFWCoreServicesPlugins.so
#3  <signal handler called>
#4  0x000014fbfaf5ce4d in std::__weak_ptr<edm::serviceregistry::ServicesManager, (__gnu_cxx::_Lock_policy)2>::__weak_ptr(std::__weak_ptr<edm::serviceregistry::ServicesManager, (__gnu_cxx::_Lock_policy)2> const&) [clone .lto_priv.0] () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02863/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-11-10-0000/lib/el8_amd64_gcc12/libFWCoreFramework.so
#5  0x000014fbfaf5f8f6 in edm::StreamSchedule::processOneStreamAsync<edm::OccurrenceTraits<edm::RunPrincipal, (edm::BranchActionType)2> >(edm::WaitingTaskHolder, edm::OccurrenceTraits<edm::RunPrincipal, (edm::BranchActionType)2>::TransitionInfoType&, edm::ServiceToken const&, bool)::{lambda()#2}::operator()() () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02863/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-11-10-0000/lib/el8_amd64_gcc12/libFWCoreFramework.so
#6  0x000014fbfaf6ed37 in tbb::detail::d1::enqueue_task<edm::StreamSchedule::processOneStreamAsync<edm::OccurrenceTraits<edm::RunPrincipal, (edm::BranchActionType)2> >(edm::WaitingTaskHolder, edm::OccurrenceTraits<edm::RunPrincipal, (edm::BranchActionType)2>::TransitionInfoType&, edm::ServiceToken const&, bool)::{lambda()#4}>::execute(tbb::detail::d1::execution_data&) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02863/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-11-10-0000/lib/el8_amd64_gcc12/libFWCoreFramework.so
#7  0x000014fbfa924b3b in tbb::detail::r1::task_dispatcher::local_wait_for_all<false, tbb::detail::r1::outermost_worker_waiter> (t=0x14fb5dace700, waiter=..., this=0x14fbf6bbbb80) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-2391c941213c757dc9a1835b31681235/tbb-v2021.9.0/src/tbb/task_dispatcher.h:322
#8  tbb::detail::r1::task_dispatcher::local_wait_for_all<tbb::detail::r1::outermost_worker_waiter> (t=0x0, waiter=..., this=0x14fbf6bbbb80) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-2391c941213c757dc9a1835b31681235/tbb-v2021.9.0/src/tbb/task_dispatcher.h:458
#9  tbb::detail::r1::arena::process (tls=..., this=<optimized out>) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-2391c941213c757dc9a1835b31681235/tbb-v2021.9.0/src/tbb/arena.cpp:137
#10 tbb::detail::r1::market::process (this=<optimized out>, j=...) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-2391c941213c757dc9a1835b31681235/tbb-v2021.9.0/src/tbb/market.cpp:599
#11 0x000014fbfa926cee in tbb::detail::r1::rml::private_worker::run (this=0x14fbf2facf00) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-2391c941213c757dc9a1835b31681235/tbb-v2021.9.0/src/tbb/private_server.cpp:271
#12 tbb::detail::r1::rml::private_worker::thread_routine (arg=0x14fbf2facf00) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-2391c941213c757dc9a1835b31681235/tbb-v2021.9.0/src/tbb/private_server.cpp:221
#13 0x000014fbf84661ca in start_thread () from /lib64/libpthread.so.0
#14 0x000014fbf80c18d3 in clone () from /lib64/libc.so.6

Thread 7 (Thread 0x14fba2bff700 (LWP 433405) "cmsRun"):
#0  0x000014fbf8190098 in nanosleep () from /lib64/libc.so.6
#1  0x000014fbf818ff9e in sleep () from /lib64/libc.so.6
#2  0x000014fbf2aba1d0 in sig_pause_for_stacktrace () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02863/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-11-10-0000/lib/el8_amd64_gcc12/pluginFWCoreServicesPlugins.so
#3  <signal handler called>
#4  0x000014fb5dd92160 in edm::stream::ProducingModuleAdaptor<SiStripZeroSuppression, edm::stream::EDProducerBase, edm::stream::EDProducerAdaptorBase>::wantsStreamLuminosityBlocks() const () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02863/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-11-10-0000/lib/el8_amd64_gcc12/pluginRecoLocalTrackerSiStripZeroSuppressionPlugins.so
#5  0x000014fbfaf5f0e9 in edm::StreamSchedule::processOneStreamAsync<edm::OccurrenceTraits<edm::RunPrincipal, (edm::BranchActionType)2> >(edm::WaitingTaskHolder, edm::OccurrenceTraits<edm::RunPrincipal, (edm::BranchActionType)2>::TransitionInfoType&, edm::ServiceToken const&, bool)::{lambda()#2}::operator()() () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02863/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-11-10-0000/lib/el8_amd64_gcc12/libFWCoreFramework.so
#6  0x000014fbfaf6ed37 in tbb::detail::d1::enqueue_task<edm::StreamSchedule::processOneStreamAsync<edm::OccurrenceTraits<edm::RunPrincipal, (edm::BranchActionType)2> >(edm::WaitingTaskHolder, edm::OccurrenceTraits<edm::RunPrincipal, (edm::BranchActionType)2>::TransitionInfoType&, edm::ServiceToken const&, bool)::{lambda()#4}>::execute(tbb::detail::d1::execution_data&) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02863/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-11-10-0000/lib/el8_amd64_gcc12/libFWCoreFramework.so
#7  0x000014fbfa924b3b in tbb::detail::r1::task_dispatcher::local_wait_for_all<false, tbb::detail::r1::outermost_worker_waiter> (t=0x14fb5dae9f00, waiter=..., this=0x14fbf6bbba80) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-2391c941213c757dc9a1835b31681235/tbb-v2021.9.0/src/tbb/task_dispatcher.h:322
#8  tbb::detail::r1::task_dispatcher::local_wait_for_all<tbb::detail::r1::outermost_worker_waiter> (t=0x0, waiter=..., this=0x14fbf6bbba80) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-2391c941213c757dc9a1835b31681235/tbb-v2021.9.0/src/tbb/task_dispatcher.h:458
#9  tbb::detail::r1::arena::process (tls=..., this=<optimized out>) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-2391c941213c757dc9a1835b31681235/tbb-v2021.9.0/src/tbb/arena.cpp:137
#10 tbb::detail::r1::market::process (this=<optimized out>, j=...) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-2391c941213c757dc9a1835b31681235/tbb-v2021.9.0/src/tbb/market.cpp:599
#11 0x000014fbfa926cee in tbb::detail::r1::rml::private_worker::run (this=0x14fbf2face80) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-2391c941213c757dc9a1835b31681235/tbb-v2021.9.0/src/tbb/private_server.cpp:271
#12 tbb::detail::r1::rml::private_worker::thread_routine (arg=0x14fbf2face80) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-2391c941213c757dc9a1835b31681235/tbb-v2021.9.0/src/tbb/private_server.cpp:221
#13 0x000014fbf84661ca in start_thread () from /lib64/libpthread.so.0
#14 0x000014fbf80c18d3 in clone () from /lib64/libc.so.6

Thread 6 (Thread 0x14fba3bfe700 (LWP 433404) "cmsRun"):
#0  0x000014fbf8190098 in nanosleep () from /lib64/libc.so.6
#1  0x000014fbf818ff9e in sleep () from /lib64/libc.so.6
#2  0x000014fbf2aba1d0 in sig_pause_for_stacktrace () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02863/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-11-10-0000/lib/el8_amd64_gcc12/pluginFWCoreServicesPlugins.so
#3  <signal handler called>
#4  0x000014fbf84701b0 in nanosleep () from /lib64/libpthread.so.0
#5  0x000014fbf2abdcaa in sig_dostack_then_abort () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02863/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-11-10-0000/lib/el8_amd64_gcc12/pluginFWCoreServicesPlugins.so
#6  <signal handler called>
#7  tbb::detail::r1::spawn (t=..., ctx=...) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-2391c941213c757dc9a1835b31681235/tbb-v2021.9.0/src/tbb/task_dispatcher.cpp:36
#8  0x000014fbfaf2e608 in tbb::detail::d1::function_task<edm::WaitingTaskHolder::doneWaiting(std::__exception_ptr::exception_ptr)::{lambda()#1}>::execute(tbb::detail::d1::execution_data&) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02863/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-11-10-0000/lib/el8_amd64_gcc12/libFWCoreFramework.so
#9  0x000014fbfa924b3b in tbb::detail::r1::task_dispatcher::local_wait_for_all<false, tbb::detail::r1::outermost_worker_waiter> (t=0x14fb5dabb000, waiter=..., this=0x14fbf6bbba00) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-2391c941213c757dc9a1835b31681235/tbb-v2021.9.0/src/tbb/task_dispatcher.h:322
#10 tbb::detail::r1::task_dispatcher::local_wait_for_all<tbb::detail::r1::outermost_worker_waiter> (t=0x0, waiter=..., this=0x14fbf6bbba00) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-2391c941213c757dc9a1835b31681235/tbb-v2021.9.0/src/tbb/task_dispatcher.h:458
#11 tbb::detail::r1::arena::process (tls=..., this=<optimized out>) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-2391c941213c757dc9a1835b31681235/tbb-v2021.9.0/src/tbb/arena.cpp:137
#12 tbb::detail::r1::market::process (this=<optimized out>, j=...) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-2391c941213c757dc9a1835b31681235/tbb-v2021.9.0/src/tbb/market.cpp:599
#13 0x000014fbfa926cee in tbb::detail::r1::rml::private_worker::run (this=0x14fbf2fad000) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-2391c941213c757dc9a1835b31681235/tbb-v2021.9.0/src/tbb/private_server.cpp:271
#14 tbb::detail::r1::rml::private_worker::thread_routine (arg=0x14fbf2fad000) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-2391c941213c757dc9a1835b31681235/tbb-v2021.9.0/src/tbb/private_server.cpp:221
#15 0x000014fbf84661ca in start_thread () from /lib64/libpthread.so.0
#16 0x000014fbf80c18d3 in clone () from /lib64/libc.so.6

Thread 5 (Thread 0x14fba45ff700 (LWP 433403) "cmsRun"):
#0  0x000014fbf8190098 in nanosleep () from /lib64/libc.so.6
#1  0x000014fbf818ff9e in sleep () from /lib64/libc.so.6
#2  0x000014fbf2aba1d0 in sig_pause_for_stacktrace () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02863/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-11-10-0000/lib/el8_amd64_gcc12/pluginFWCoreServicesPlugins.so
#3  <signal handler called>
#4  tbb::detail::r1::machine_time_stamp () at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-2391c941213c757dc9a1835b31681235/tbb-v2021.9.0/src/tbb/scheduler_common.h:190
#5  tbb::detail::r1::prolonged_pause_impl () at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-2391c941213c757dc9a1835b31681235/tbb-v2021.9.0/src/tbb/scheduler_common.h:203
#6  tbb::detail::r1::prolonged_pause () at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-2391c941213c757dc9a1835b31681235/tbb-v2021.9.0/src/tbb/scheduler_common.h:234
#7  tbb::detail::r1::stealing_loop_backoff::pause (this=0x14fba45f8ef8) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-2391c941213c757dc9a1835b31681235/tbb-v2021.9.0/src/tbb/scheduler_common.h:263
#8  tbb::detail::r1::waiter_base::pause (this=0x14fba45f8ef0) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-2391c941213c757dc9a1835b31681235/tbb-v2021.9.0/src/tbb/waiters.h:35
#9  tbb::detail::r1::outermost_worker_waiter::pause (this=0x14fba45f8ef0) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-2391c941213c757dc9a1835b31681235/tbb-v2021.9.0/src/tbb/waiters.h:69
#10 tbb::detail::r1::task_dispatcher::receive_or_steal_task<true, tbb::detail::r1::outermost_worker_waiter> (this=<optimized out>, tls=..., ed=..., waiter=..., isolation=<optimized out>, fifo_allowed=<optimized out>, critical_allowed=<optimized out>) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-2391c941213c757dc9a1835b31681235/tbb-v2021.9.0/src/tbb/task_dispatcher.h:231
#11 0x000014fbfa924cb2 in tbb::detail::r1::task_dispatcher::local_wait_for_all<false, tbb::detail::r1::outermost_worker_waiter> (t=0x0, waiter=..., this=0x14fbf6bbbc00) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-2391c941213c757dc9a1835b31681235/tbb-v2021.9.0/src/tbb/task_dispatcher.h:350
#12 tbb::detail::r1::task_dispatcher::local_wait_for_all<tbb::detail::r1::outermost_worker_waiter> (t=0x0, waiter=..., this=0x14fbf6bbbc00) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-2391c941213c757dc9a1835b31681235/tbb-v2021.9.0/src/tbb/task_dispatcher.h:458
#13 tbb::detail::r1::arena::process (tls=..., this=<optimized out>) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-2391c941213c757dc9a1835b31681235/tbb-v2021.9.0/src/tbb/arena.cpp:137
#14 tbb::detail::r1::market::process (this=<optimized out>, j=...) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-2391c941213c757dc9a1835b31681235/tbb-v2021.9.0/src/tbb/market.cpp:599
#15 0x000014fbfa926cee in tbb::detail::r1::rml::private_worker::run (this=0x14fbf2facf80) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-2391c941213c757dc9a1835b31681235/tbb-v2021.9.0/src/tbb/private_server.cpp:271
#16 tbb::detail::r1::rml::private_worker::thread_routine (arg=0x14fbf2facf80) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-2391c941213c757dc9a1835b31681235/tbb-v2021.9.0/src/tbb/private_server.cpp:221
#17 0x000014fbf84661ca in start_thread () from /lib64/libpthread.so.0
#18 0x000014fbf80c18d3 in clone () from /lib64/libc.so.6

Thread 4 (Thread 0x14fba559f700 (LWP 433402) "cmsRun"):
#0  0x000014fbf8190098 in nanosleep () from /lib64/libc.so.6
#1  0x000014fbf818ff9e in sleep () from /lib64/libc.so.6
#2  0x000014fbf2aba1d0 in sig_pause_for_stacktrace () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02863/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-11-10-0000/lib/el8_amd64_gcc12/pluginFWCoreServicesPlugins.so
#3  <signal handler called>
#4  0x000014fbfb1d5fd5 in edm::WaitingTaskList::announce() () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02863/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-11-10-0000/lib/el8_amd64_gcc12/libFWCoreConcurrency.so
#5  0x000014fbfaf62630 in edm::Worker::doWorkNoPrefetchingAsync<edm::OccurrenceTraits<edm::RunPrincipal, (edm::BranchActionType)2> >(edm::WaitingTaskHolder, edm::OccurrenceTraits<edm::RunPrincipal, (edm::BranchActionType)2>::TransitionInfoType const&, edm::ServiceToken const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::RunPrincipal, (edm::BranchActionType)2>::Context const*)::{lambda()#1}::operator()() const () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02863/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-11-10-0000/lib/el8_amd64_gcc12/libFWCoreFramework.so
#6  0x000014fbfaf6e8e9 in tbb::detail::d1::function_task<edm::Worker::doWorkNoPrefetchingAsync<edm::OccurrenceTraits<edm::RunPrincipal, (edm::BranchActionType)2> >(edm::WaitingTaskHolder, edm::OccurrenceTraits<edm::RunPrincipal, (edm::BranchActionType)2>::TransitionInfoType const&, edm::ServiceToken const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::RunPrincipal, (edm::BranchActionType)2>::Context const*)::{lambda()#1}>::execute(tbb::detail::d1::execution_data&) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02863/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-11-10-0000/lib/el8_amd64_gcc12/libFWCoreFramework.so
#7  0x000014fbfa924b3b in tbb::detail::r1::task_dispatcher::local_wait_for_all<false, tbb::detail::r1::outermost_worker_waiter> (t=0x14fb5d9b7600, waiter=..., this=0x14fbf6bbb980) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-2391c941213c757dc9a1835b31681235/tbb-v2021.9.0/src/tbb/task_dispatcher.h:322
#8  tbb::detail::r1::task_dispatcher::local_wait_for_all<tbb::detail::r1::outermost_worker_waiter> (t=0x0, waiter=..., this=0x14fbf6bbb980) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-2391c941213c757dc9a1835b31681235/tbb-v2021.9.0/src/tbb/task_dispatcher.h:458
#9  tbb::detail::r1::arena::process (tls=..., this=<optimized out>) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-2391c941213c757dc9a1835b31681235/tbb-v2021.9.0/src/tbb/arena.cpp:137
#10 tbb::detail::r1::market::process (this=<optimized out>, j=...) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-2391c941213c757dc9a1835b31681235/tbb-v2021.9.0/src/tbb/market.cpp:599
#11 0x000014fbfa926cee in tbb::detail::r1::rml::private_worker::run (this=0x14fbf2fad100) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-2391c941213c757dc9a1835b31681235/tbb-v2021.9.0/src/tbb/private_server.cpp:271
#12 tbb::detail::r1::rml::private_worker::thread_routine (arg=0x14fbf2fad100) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-2391c941213c757dc9a1835b31681235/tbb-v2021.9.0/src/tbb/private_server.cpp:221
#13 0x000014fbf84661ca in start_thread () from /lib64/libpthread.so.0
#14 0x000014fbf80c18d3 in clone () from /lib64/libc.so.6

Thread 3 (Thread 0x14fba5fa0700 (LWP 433401) "cmsRun"):
#0  0x000014fbf8190098 in nanosleep () from /lib64/libc.so.6
#1  0x000014fbf818ff9e in sleep () from /lib64/libc.so.6
#2  0x000014fbf2aba1d0 in sig_pause_for_stacktrace () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02863/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-11-10-0000/lib/el8_amd64_gcc12/pluginFWCoreServicesPlugins.so
#3  <signal handler called>
#4  0x000014fbfaf54bca in std::_Sp_counted_base<(__gnu_cxx::_Lock_policy)2>::_M_weak_release() () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02863/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-11-10-0000/lib/el8_amd64_gcc12/libFWCoreFramework.so
#5  0x000014fbfaf5f053 in edm::StreamSchedule::processOneStreamAsync<edm::OccurrenceTraits<edm::RunPrincipal, (edm::BranchActionType)2> >(edm::WaitingTaskHolder, edm::OccurrenceTraits<edm::RunPrincipal, (edm::BranchActionType)2>::TransitionInfoType&, edm::ServiceToken const&, bool)::{lambda()#2}::operator()() () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02863/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-11-10-0000/lib/el8_amd64_gcc12/libFWCoreFramework.so
#6  0x000014fbfaf6ed37 in tbb::detail::d1::enqueue_task<edm::StreamSchedule::processOneStreamAsync<edm::OccurrenceTraits<edm::RunPrincipal, (edm::BranchActionType)2> >(edm::WaitingTaskHolder, edm::OccurrenceTraits<edm::RunPrincipal, (edm::BranchActionType)2>::TransitionInfoType&, edm::ServiceToken const&, bool)::{lambda()#4}>::execute(tbb::detail::d1::execution_data&) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02863/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-11-10-0000/lib/el8_amd64_gcc12/libFWCoreFramework.so
#7  0x000014fbfa924b3b in tbb::detail::r1::task_dispatcher::local_wait_for_all<false, tbb::detail::r1::outermost_worker_waiter> (t=0x14fb5daea900, waiter=..., this=0x14fbf6bbbb00) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-2391c941213c757dc9a1835b31681235/tbb-v2021.9.0/src/tbb/task_dispatcher.h:322
#8  tbb::detail::r1::task_dispatcher::local_wait_for_all<tbb::detail::r1::outermost_worker_waiter> (t=0x0, waiter=..., this=0x14fbf6bbbb00) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-2391c941213c757dc9a1835b31681235/tbb-v2021.9.0/src/tbb/task_dispatcher.h:458
#9  tbb::detail::r1::arena::process (tls=..., this=<optimized out>) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-2391c941213c757dc9a1835b31681235/tbb-v2021.9.0/src/tbb/arena.cpp:137
#10 tbb::detail::r1::market::process (this=<optimized out>, j=...) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-2391c941213c757dc9a1835b31681235/tbb-v2021.9.0/src/tbb/market.cpp:599
#11 0x000014fbfa926cee in tbb::detail::r1::rml::private_worker::run (this=0x14fbf2fad080) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-2391c941213c757dc9a1835b31681235/tbb-v2021.9.0/src/tbb/private_server.cpp:271
#12 tbb::detail::r1::rml::private_worker::thread_routine (arg=0x14fbf2fad080) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-2391c941213c757dc9a1835b31681235/tbb-v2021.9.0/src/tbb/private_server.cpp:221
#13 0x000014fbf84661ca in start_thread () from /lib64/libpthread.so.0
#14 0x000014fbf80c18d3 in clone () from /lib64/libc.so.6

Thread 2 (Thread 0x14fbceba9700 (LWP 433308) "cmsRun"):
#0  0x000014fbf84706a2 in waitpid () from /lib64/libpthread.so.0
#1  0x000014fbf2aba327 in edm::service::cmssw_stacktrace_fork() () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02863/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-11-10-0000/lib/el8_amd64_gcc12/pluginFWCoreServicesPlugins.so
#2  0x000014fbf2abd63a in edm::service::InitRootHandlers::stacktraceHelperThread() () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02863/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-11-10-0000/lib/el8_amd64_gcc12/pluginFWCoreServicesPlugins.so
#3  0x000014fbf8ad8a73 in std::execute_native_thread_routine (__p=0x14fbd2196ea0) at ../../../../../libstdc++-v3/src/c++11/thread.cc:82
#4  0x000014fbf84661ca in start_thread () from /lib64/libpthread.so.0
#5  0x000014fbf80c18d3 in clone () from /lib64/libc.so.6

Thread 1 (Thread 0x14fbfa0f7680 (LWP 432190) "cmsRun"):
#0  0x000014fbf81baac1 in poll () from /lib64/libc.so.6
#1  0x000014fbf2abd857 in edm::service::InitRootHandlers::stacktraceFromThread() () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02863/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-11-10-0000/lib/el8_amd64_gcc12/pluginFWCoreServicesPlugins.so
#2  0x000014fbf2abda54 in sig_dostack_then_abort () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02863/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-11-10-0000/lib/el8_amd64_gcc12/pluginFWCoreServicesPlugins.so
#3  <signal handler called>
#4  tbb::detail::r1::spawn (t=..., ctx=...) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-2391c941213c757dc9a1835b31681235/tbb-v2021.9.0/src/tbb/task_dispatcher.cpp:36
#5  0x000014fbfaf2e608 in tbb::detail::d1::function_task<edm::WaitingTaskHolder::doneWaiting(std::__exception_ptr::exception_ptr)::{lambda()#1}>::execute(tbb::detail::d1::execution_data&) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02863/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-11-10-0000/lib/el8_amd64_gcc12/libFWCoreFramework.so
#6  0x000014fbfa92d3e1 in tbb::detail::r1::task_dispatcher::local_wait_for_all<false, tbb::detail::r1::external_waiter> (waiter=..., t=<optimized out>, this=0x14fbf6bbb900) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-2391c941213c757dc9a1835b31681235/tbb-v2021.9.0/src/tbb/task_dispatcher.h:322
#7  tbb::detail::r1::task_dispatcher::local_wait_for_all<tbb::detail::r1::external_waiter> (waiter=..., t=<optimized out>, this=0x14fbf6bbb900) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-2391c941213c757dc9a1835b31681235/tbb-v2021.9.0/src/tbb/task_dispatcher.h:458
#8  tbb::detail::r1::task_dispatcher::execute_and_wait (t=<optimized out>, wait_ctx=..., w_ctx=...) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-2391c941213c757dc9a1835b31681235/tbb-v2021.9.0/src/tbb/task_dispatcher.cpp:168
#9  0x000014fbfaf4450b in edm::FinalWaitingTask::wait() () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02863/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-11-10-0000/lib/el8_amd64_gcc12/libFWCoreFramework.so
#10 0x000014fbfaf51196 in edm::EventProcessor::endUnfinishedRun(bool) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02863/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-11-10-0000/lib/el8_amd64_gcc12/libFWCoreFramework.so
#11 0x000014fbfaf524ed in edm::EventProcessor::runToCompletion() () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02863/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-11-10-0000/lib/el8_amd64_gcc12/libFWCoreFramework.so
#12 0x000000000040840c in tbb::detail::d1::task_arena_function<main::{lambda()#1}::operator()() const::{lambda()#1}, void>::operator()() const ()
#13 0x000014fbfa9199ad in tbb::detail::r1::task_arena_impl::execute (ta=..., d=...) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-2391c941213c757dc9a1835b31681235/tbb-v2021.9.0/src/tbb/arena.cpp:688
#14 0x000000000040a0f2 in main::{lambda()#1}::operator()() const ()
#15 0x0000000000405100 in main ()

Current Modules:

Module: none (crashed)
Module: none
Module: none
Module: TrackProducerWithSCAssociation:uncleanedOnlyCkfInOutTracksFromConversions
Module: none
Module: none
Module: none
Module: none
Module: none
Module: none
Module: TrackProducerWithSCAssociation:uncleanedOnlyCkfInOutTracksFromConversions
Module: none
Module: none
Module: none
Module: none
Module: none
Module: none

Also here threads 6 and 1 show sig_dostack_then_abort


Thread 10 (Thread 0x14dc447fc700 (LWP 433430) "cmsRun"):
#0  0x000014dc9b990098 in nanosleep () from /lib64/libc.so.6
#1  0x000014dc9b98ff9e in sleep () from /lib64/libc.so.6
#2  0x000014dc961c31d0 in sig_pause_for_stacktrace () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02863/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-11-10-0000/lib/el8_amd64_gcc12/pluginFWCoreServicesPlugins.so
#3  <signal handler called>
#4  0x000014dc9b8c141d in syscall () from /lib64/libc.so.6
#5  0x000014dc9d98dfd2 in tbb::detail::r1::futex_wait (comparand=2, futex=0x14dc966b5f24) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-2391c941213c757dc9a1835b31681235/tbb-v2021.9.0/src/tbb/semaphore.h:100
#6  tbb::detail::r1::binary_semaphore::P (this=0x14dc966b5f24) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-2391c941213c757dc9a1835b31681235/tbb-v2021.9.0/src/tbb/semaphore.h:253
#7  tbb::detail::r1::rml::internal::thread_monitor::wait (this=0x14dc966b5f20) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-2391c941213c757dc9a1835b31681235/tbb-v2021.9.0/src/tbb/rml_thread_monitor.h:235
#8  tbb::detail::r1::rml::private_worker::run (this=0x14dc966b5f00) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-2391c941213c757dc9a1835b31681235/tbb-v2021.9.0/src/tbb/private_server.cpp:273
#9  tbb::detail::r1::rml::private_worker::thread_routine (arg=0x14dc966b5f00) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-2391c941213c757dc9a1835b31681235/tbb-v2021.9.0/src/tbb/private_server.cpp:221
#10 0x000014dc9bc661ca in start_thread () from /lib64/libpthread.so.0
#11 0x000014dc9b8c18d3 in clone () from /lib64/libc.so.6

Thread 9 (Thread 0x14dc451fd700 (LWP 433429) "cmsRun"):
#0  0x000014dc9b990098 in nanosleep () from /lib64/libc.so.6
#1  0x000014dc9b98ff9e in sleep () from /lib64/libc.so.6
#2  0x000014dc961c31d0 in sig_pause_for_stacktrace () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02863/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-11-10-0000/lib/el8_amd64_gcc12/pluginFWCoreServicesPlugins.so
#3  <signal handler called>
#4  0x000014dc9e56ed20 in tbb::detail::d1::enqueue_task<edm::StreamSchedule::processOneStreamAsync<edm::OccurrenceTraits<edm::RunPrincipal, (edm::BranchActionType)2> >(edm::WaitingTaskHolder, edm::OccurrenceTraits<edm::RunPrincipal, (edm::BranchActionType)2>::TransitionInfoType&, edm::ServiceToken const&, bool)::{lambda()#4}>::execute(tbb::detail::d1::execution_data&) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02863/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-11-10-0000/lib/el8_amd64_gcc12/libFWCoreFramework.so
#5  0x000014dc9d98bb3b in tbb::detail::r1::task_dispatcher::local_wait_for_all<false, tbb::detail::r1::outermost_worker_waiter> (t=0x14dc9a279700, waiter=..., this=0x14dc9a3bbc00) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-2391c941213c757dc9a1835b31681235/tbb-v2021.9.0/src/tbb/task_dispatcher.h:322
#6  tbb::detail::r1::task_dispatcher::local_wait_for_all<tbb::detail::r1::outermost_worker_waiter> (t=0x0, waiter=..., this=0x14dc9a3bbc00) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-2391c941213c757dc9a1835b31681235/tbb-v2021.9.0/src/tbb/task_dispatcher.h:458
#7  tbb::detail::r1::arena::process (tls=..., this=<optimized out>) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-2391c941213c757dc9a1835b31681235/tbb-v2021.9.0/src/tbb/arena.cpp:137
#8  tbb::detail::r1::market::process (this=<optimized out>, j=...) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-2391c941213c757dc9a1835b31681235/tbb-v2021.9.0/src/tbb/market.cpp:599
#9  0x000014dc9d98dcee in tbb::detail::r1::rml::private_worker::run (this=0x14dc966b5e80) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-2391c941213c757dc9a1835b31681235/tbb-v2021.9.0/src/tbb/private_server.cpp:271
#10 tbb::detail::r1::rml::private_worker::thread_routine (arg=0x14dc966b5e80) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-2391c941213c757dc9a1835b31681235/tbb-v2021.9.0/src/tbb/private_server.cpp:221
#11 0x000014dc9bc661ca in start_thread () from /lib64/libpthread.so.0
#12 0x000014dc9b8c18d3 in clone () from /lib64/libc.so.6

Thread 8 (Thread 0x14dc453fe700 (LWP 433428) "cmsRun"):
#0  0x000014dc9b9c7307 in epoll_wait () from /lib64/libc.so.6
#1  0x000014dc95d53b62 in XrdSys::IOEvents::PollE::Begin(XrdSysSemaphore*, int&, char const**) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02863/el8_amd64_gcc12/cms/cmssw-patch/CMSSW_14_1_X_2024-11-13-1100/external/el8_amd64_gcc12/lib/libXrdUtils.so.3
#2  0x000014dc95d4facd in XrdSys::IOEvents::BootStrap::Start(void*) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02863/el8_amd64_gcc12/cms/cmssw-patch/CMSSW_14_1_X_2024-11-13-1100/external/el8_amd64_gcc12/lib/libXrdUtils.so.3
#3  0x000014dc95d58ef7 in XrdSysThread_Xeq () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02863/el8_amd64_gcc12/cms/cmssw-patch/CMSSW_14_1_X_2024-11-13-1100/external/el8_amd64_gcc12/lib/libXrdUtils.so.3
#4  0x000014dc9bc661ca in start_thread () from /lib64/libpthread.so.0
#5  0x000014dc9b8c18d3 in clone () from /lib64/libc.so.6

Thread 7 (Thread 0x14dc46fff700 (LWP 433427) "cmsRun"):
#0  0x000014dc9b9baac1 in poll () from /lib64/libc.so.6
#1  0x000014dc961c6857 in edm::service::InitRootHandlers::stacktraceFromThread() () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02863/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-11-10-0000/lib/el8_amd64_gcc12/pluginFWCoreServicesPlugins.so
#2  0x000014dc961c6a54 in sig_dostack_then_abort () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02863/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-11-10-0000/lib/el8_amd64_gcc12/pluginFWCoreServicesPlugins.so
#3  <signal handler called>
#4  tbb::detail::r1::spawn (t=..., ctx=...) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-2391c941213c757dc9a1835b31681235/tbb-v2021.9.0/src/tbb/task_dispatcher.cpp:36
#5  0x000014dc9e52e608 in tbb::detail::d1::function_task<edm::WaitingTaskHolder::doneWaiting(std::__exception_ptr::exception_ptr)::{lambda()#1}>::execute(tbb::detail::d1::execution_data&) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02863/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-11-10-0000/lib/el8_amd64_gcc12/libFWCoreFramework.so
#6  0x000014dc9d98bb3b in tbb::detail::r1::task_dispatcher::local_wait_for_all<false, tbb::detail::r1::outermost_worker_waiter> (t=0x14dc9a279900, waiter=..., this=0x14dc9a3bbb80) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-2391c941213c757dc9a1835b31681235/tbb-v2021.9.0/src/tbb/task_dispatcher.h:322
#7  tbb::detail::r1::task_dispatcher::local_wait_for_all<tbb::detail::r1::outermost_worker_waiter> (t=0x0, waiter=..., this=0x14dc9a3bbb80) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-2391c941213c757dc9a1835b31681235/tbb-v2021.9.0/src/tbb/task_dispatcher.h:458
#8  tbb::detail::r1::arena::process (tls=..., this=<optimized out>) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-2391c941213c757dc9a1835b31681235/tbb-v2021.9.0/src/tbb/arena.cpp:137
#9  tbb::detail::r1::market::process (this=<optimized out>, j=...) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-2391c941213c757dc9a1835b31681235/tbb-v2021.9.0/src/tbb/market.cpp:599
#10 0x000014dc9d98dcee in tbb::detail::r1::rml::private_worker::run (this=0x14dc966b6000) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-2391c941213c757dc9a1835b31681235/tbb-v2021.9.0/src/tbb/private_server.cpp:271
#11 tbb::detail::r1::rml::private_worker::thread_routine (arg=0x14dc966b6000) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-2391c941213c757dc9a1835b31681235/tbb-v2021.9.0/src/tbb/private_server.cpp:221
#12 0x000014dc9bc661ca in start_thread () from /lib64/libpthread.so.0
#13 0x000014dc9b8c18d3 in clone () from /lib64/libc.so.6

Thread 6 (Thread 0x14dc45fff700 (LWP 433426) "cmsRun"):
#0  0x000014dc9b990098 in nanosleep () from /lib64/libc.so.6
#1  0x000014dc9b98ff9e in sleep () from /lib64/libc.so.6
#2  0x000014dc961c31d0 in sig_pause_for_stacktrace () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02863/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-11-10-0000/lib/el8_amd64_gcc12/pluginFWCoreServicesPlugins.so
#3  <signal handler called>
#4  0x000014dc9b8c141d in syscall () from /lib64/libc.so.6
#5  0x000014dc9d98dfd2 in tbb::detail::r1::futex_wait (comparand=2, futex=0x14dc966b5e24) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-2391c941213c757dc9a1835b31681235/tbb-v2021.9.0/src/tbb/semaphore.h:100
#6  tbb::detail::r1::binary_semaphore::P (this=0x14dc966b5e24) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-2391c941213c757dc9a1835b31681235/tbb-v2021.9.0/src/tbb/semaphore.h:253
#7  tbb::detail::r1::rml::internal::thread_monitor::wait (this=0x14dc966b5e20) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-2391c941213c757dc9a1835b31681235/tbb-v2021.9.0/src/tbb/rml_thread_monitor.h:235
#8  tbb::detail::r1::rml::private_worker::run (this=0x14dc966b5e00) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-2391c941213c757dc9a1835b31681235/tbb-v2021.9.0/src/tbb/private_server.cpp:273
#9  tbb::detail::r1::rml::private_worker::thread_routine (arg=0x14dc966b5e00) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-2391c941213c757dc9a1835b31681235/tbb-v2021.9.0/src/tbb/private_server.cpp:221
#10 0x000014dc9bc661ca in start_thread () from /lib64/libpthread.so.0
#11 0x000014dc9b8c18d3 in clone () from /lib64/libc.so.6

Thread 5 (Thread 0x14dc47d6b700 (LWP 433425) "cmsRun"):
#0  0x000014dc9b990098 in nanosleep () from /lib64/libc.so.6
#1  0x000014dc9b98ff9e in sleep () from /lib64/libc.so.6
#2  0x000014dc961c31d0 in sig_pause_for_stacktrace () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02863/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-11-10-0000/lib/el8_amd64_gcc12/pluginFWCoreServicesPlugins.so
#3  <signal handler called>
#4  0x000014dc9e55f23f in edm::StreamSchedule::processOneStreamAsync<edm::OccurrenceTraits<edm::RunPrincipal, (edm::BranchActionType)2> >(edm::WaitingTaskHolder, edm::OccurrenceTraits<edm::RunPrincipal, (edm::BranchActionType)2>::TransitionInfoType&, edm::ServiceToken const&, bool)::{lambda()#2}::operator()() () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02863/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-11-10-0000/lib/el8_amd64_gcc12/libFWCoreFramework.so
#5  0x000014dc9e56ed37 in tbb::detail::d1::enqueue_task<edm::StreamSchedule::processOneStreamAsync<edm::OccurrenceTraits<edm::RunPrincipal, (edm::BranchActionType)2> >(edm::WaitingTaskHolder, edm::OccurrenceTraits<edm::RunPrincipal, (edm::BranchActionType)2>::TransitionInfoType&, edm::ServiceToken const&, bool)::{lambda()#4}>::execute(tbb::detail::d1::execution_data&) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02863/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-11-10-0000/lib/el8_amd64_gcc12/libFWCoreFramework.so
#6  0x000014dc9d98bb3b in tbb::detail::r1::task_dispatcher::local_wait_for_all<false, tbb::detail::r1::outermost_worker_waiter> (t=0x14dc9a279600, waiter=..., this=0x14dc9a3bb980) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-2391c941213c757dc9a1835b31681235/tbb-v2021.9.0/src/tbb/task_dispatcher.h:322
#7  tbb::detail::r1::task_dispatcher::local_wait_for_all<tbb::detail::r1::outermost_worker_waiter> (t=0x0, waiter=..., this=0x14dc9a3bb980) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-2391c941213c757dc9a1835b31681235/tbb-v2021.9.0/src/tbb/task_dispatcher.h:458
#8  tbb::detail::r1::arena::process (tls=..., this=<optimized out>) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-2391c941213c757dc9a1835b31681235/tbb-v2021.9.0/src/tbb/arena.cpp:137
#9  tbb::detail::r1::market::process (this=<optimized out>, j=...) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-2391c941213c757dc9a1835b31681235/tbb-v2021.9.0/src/tbb/market.cpp:599
#10 0x000014dc9d98dcee in tbb::detail::r1::rml::private_worker::run (this=0x14dc966b5f80) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-2391c941213c757dc9a1835b31681235/tbb-v2021.9.0/src/tbb/private_server.cpp:271
#11 tbb::detail::r1::rml::private_worker::thread_routine (arg=0x14dc966b5f80) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-2391c941213c757dc9a1835b31681235/tbb-v2021.9.0/src/tbb/private_server.cpp:221
#12 0x000014dc9bc661ca in start_thread () from /lib64/libpthread.so.0
#13 0x000014dc9b8c18d3 in clone () from /lib64/libc.so.6

Thread 4 (Thread 0x14dc48c6c700 (LWP 433424) "cmsRun"):
#0  0x000014dc9b990098 in nanosleep () from /lib64/libc.so.6
#1  0x000014dc9b98ff9e in sleep () from /lib64/libc.so.6
#2  0x000014dc961c31d0 in sig_pause_for_stacktrace () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02863/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-11-10-0000/lib/el8_amd64_gcc12/pluginFWCoreServicesPlugins.so
#3  <signal handler called>
#4  0x000014dc9b8c141d in syscall () from /lib64/libc.so.6
#5  0x000014dc9d98dfd2 in tbb::detail::r1::futex_wait (comparand=2, futex=0x14dc966b6124) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-2391c941213c757dc9a1835b31681235/tbb-v2021.9.0/src/tbb/semaphore.h:100
#6  tbb::detail::r1::binary_semaphore::P (this=0x14dc966b6124) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-2391c941213c757dc9a1835b31681235/tbb-v2021.9.0/src/tbb/semaphore.h:253
#7  tbb::detail::r1::rml::internal::thread_monitor::wait (this=0x14dc966b6120) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-2391c941213c757dc9a1835b31681235/tbb-v2021.9.0/src/tbb/rml_thread_monitor.h:235
#8  tbb::detail::r1::rml::private_worker::run (this=0x14dc966b6100) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-2391c941213c757dc9a1835b31681235/tbb-v2021.9.0/src/tbb/private_server.cpp:273
#9  tbb::detail::r1::rml::private_worker::thread_routine (arg=0x14dc966b6100) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-2391c941213c757dc9a1835b31681235/tbb-v2021.9.0/src/tbb/private_server.cpp:221
#10 0x000014dc9bc661ca in start_thread () from /lib64/libpthread.so.0
#11 0x000014dc9b8c18d3 in clone () from /lib64/libc.so.6

Thread 3 (Thread 0x14dc4966d700 (LWP 433423) "cmsRun"):
#0  0x000014dc9b990098 in nanosleep () from /lib64/libc.so.6
#1  0x000014dc9b98ff9e in sleep () from /lib64/libc.so.6
#2  0x000014dc961c31d0 in sig_pause_for_stacktrace () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02863/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-11-10-0000/lib/el8_amd64_gcc12/pluginFWCoreServicesPlugins.so
#3  <signal handler called>
#4  0x000014dc9b8c141d in syscall () from /lib64/libc.so.6
#5  0x000014dc9d98dfd2 in tbb::detail::r1::futex_wait (comparand=2, futex=0x14dc966b60a4) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-2391c941213c757dc9a1835b31681235/tbb-v2021.9.0/src/tbb/semaphore.h:100
#6  tbb::detail::r1::binary_semaphore::P (this=0x14dc966b60a4) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-2391c941213c757dc9a1835b31681235/tbb-v2021.9.0/src/tbb/semaphore.h:253
#7  tbb::detail::r1::rml::internal::thread_monitor::wait (this=0x14dc966b60a0) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-2391c941213c757dc9a1835b31681235/tbb-v2021.9.0/src/tbb/rml_thread_monitor.h:235
#8  tbb::detail::r1::rml::private_worker::run (this=0x14dc966b6080) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-2391c941213c757dc9a1835b31681235/tbb-v2021.9.0/src/tbb/private_server.cpp:273
#9  tbb::detail::r1::rml::private_worker::thread_routine (arg=0x14dc966b6080) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-2391c941213c757dc9a1835b31681235/tbb-v2021.9.0/src/tbb/private_server.cpp:221
#10 0x000014dc9bc661ca in start_thread () from /lib64/libpthread.so.0
#11 0x000014dc9b8c18d3 in clone () from /lib64/libc.so.6

Thread 2 (Thread 0x14dc71d03700 (LWP 433316) "cmsRun"):
#0  0x000014dc9bc706a2 in waitpid () from /lib64/libpthread.so.0
#1  0x000014dc961c3327 in edm::service::cmssw_stacktrace_fork() () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02863/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-11-10-0000/lib/el8_amd64_gcc12/pluginFWCoreServicesPlugins.so
#2  0x000014dc961c663a in edm::service::InitRootHandlers::stacktraceHelperThread() () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02863/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-11-10-0000/lib/el8_amd64_gcc12/pluginFWCoreServicesPlugins.so
#3  0x000014dc9c2d8a73 in std::execute_native_thread_routine (__p=0x14dc752fceb0) at ../../../../../libstdc++-v3/src/c++11/thread.cc:82
#4  0x000014dc9bc661ca in start_thread () from /lib64/libpthread.so.0
#5  0x000014dc9b8c18d3 in clone () from /lib64/libc.so.6

Thread 1 (Thread 0x14dc9d8c2680 (LWP 432468) "cmsRun"):
#0  0x000014dc9b990098 in nanosleep () from /lib64/libc.so.6
#1  0x000014dc9b98ff9e in sleep () from /lib64/libc.so.6
#2  0x000014dc961c31d0 in sig_pause_for_stacktrace () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02863/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-11-10-0000/lib/el8_amd64_gcc12/pluginFWCoreServicesPlugins.so
#3  <signal handler called>
#4  0x000014dc9e55f23f in edm::StreamSchedule::processOneStreamAsync<edm::OccurrenceTraits<edm::RunPrincipal, (edm::BranchActionType)2> >(edm::WaitingTaskHolder, edm::OccurrenceTraits<edm::RunPrincipal, (edm::BranchActionType)2>::TransitionInfoType&, edm::ServiceToken const&, bool)::{lambda()#2}::operator()() () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02863/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-11-10-0000/lib/el8_amd64_gcc12/libFWCoreFramework.so
#5  0x000014dc9e56edad in tbb::detail::d1::function_task<edm::StreamSchedule::processOneStreamAsync<edm::OccurrenceTraits<edm::RunPrincipal, (edm::BranchActionType)2> >(edm::WaitingTaskHolder, edm::OccurrenceTraits<edm::RunPrincipal, (edm::BranchActionType)2>::TransitionInfoType&, edm::ServiceToken const&, bool)::{lambda()#3}>::execute(tbb::detail::d1::execution_data&) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02863/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-11-10-0000/lib/el8_amd64_gcc12/libFWCoreFramework.so
#6  0x000014dc9d9943e1 in tbb::detail::r1::task_dispatcher::local_wait_for_all<false, tbb::detail::r1::external_waiter> (waiter=..., t=<optimized out>, this=0x14dc9a3bb900) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-2391c941213c757dc9a1835b31681235/tbb-v2021.9.0/src/tbb/task_dispatcher.h:322
#7  tbb::detail::r1::task_dispatcher::local_wait_for_all<tbb::detail::r1::external_waiter> (waiter=..., t=<optimized out>, this=0x14dc9a3bb900) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-2391c941213c757dc9a1835b31681235/tbb-v2021.9.0/src/tbb/task_dispatcher.h:458
#8  tbb::detail::r1::task_dispatcher::execute_and_wait (t=<optimized out>, wait_ctx=..., w_ctx=...) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-2391c941213c757dc9a1835b31681235/tbb-v2021.9.0/src/tbb/task_dispatcher.cpp:168
#9  0x000014dc9e54450b in edm::FinalWaitingTask::wait() () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02863/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-11-10-0000/lib/el8_amd64_gcc12/libFWCoreFramework.so
#10 0x000014dc9e551196 in edm::EventProcessor::endUnfinishedRun(bool) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02863/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-11-10-0000/lib/el8_amd64_gcc12/libFWCoreFramework.so
#11 0x000014dc9e5524ed in edm::EventProcessor::runToCompletion() () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02863/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-11-10-0000/lib/el8_amd64_gcc12/libFWCoreFramework.so
#12 0x000000000040840c in tbb::detail::d1::task_arena_function<main::{lambda()#1}::operator()() const::{lambda()#1}, void>::operator()() const ()
#13 0x000014dc9d9809ad in tbb::detail::r1::task_arena_impl::execute (ta=..., d=...) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-2391c941213c757dc9a1835b31681235/tbb-v2021.9.0/src/tbb/arena.cpp:688
#14 0x000000000040a0f2 in main::{lambda()#1}::operator()() const ()
#15 0x0000000000405100 in main ()

Current Modules:

Module: none

@makortel
Copy link
Contributor Author

The crash itself seems to happen always in tbb::detail::r1::spawn(). I noticed the latest oneTBB release is 2022.0.0 (released 2 weeks ago), whereas we are in 2021.9.0. Going through the intermediate release notes I didn't spot anything relevant though, except maybe (link)

Fixed the missed signal for thread request for enqueue operation.

(but I have hard time understanding what exactly that means).

@smuzaffar Maybe we could consider updating oneTBB once master moves to 15_0_X?

@makortel
Copy link
Contributor Author

Occurred in #46769 (comment)

Begin processing the 100th record. Run 355380, Event 26785259, LumiSection 19 on stream 4 at 26-Nov-2024 03:14:38.922 CET
%MSG-s NewIOV:  PostProcessPath 26-Nov-2024 03:14:44 CET  Run: 355380 Lumi: 19 Stream
Fetched new IOV for 'BeamSpotOnlineHLT' request interval [ 1526345477652503 , 1526349772619783 ] new range [ 1526345477652503 , 1526349772619782 ] #entries 49
%MSG
%MSG-s NewIOV:  PostProcessPath 26-Nov-2024 03:14:44 CET  Run: 355380 Lumi: 19 Stream
Fetched new IOV for 'BeamSpotOnlineLegacy' request interval [ 1526345477652504 , 1526349772619785 ] new range [ 1526345477652504 , 1526349772619784 ] #entries 49
%MSG

<cut>

Thread 9 (Thread 0x151a725ff700 (LWP 303308) "cmsRun"):
#0  0x0000151acbb90098 in nanosleep () from /lib64/libc.so.6
#1  0x0000151acbb8ff9e in sleep () from /lib64/libc.so.6
#2  0x0000151ac45b9360 in sig_pause_for_stacktrace () from /data/cmsbld/jenkins/workspace/ib-run-pr-tests/CMSSW_15_0_X_2024-11-25-1100/lib/el8_amd64_gcc12/pluginFWCoreServicesPlugins.so
#3  <signal handler called>
#4  0x0000151acbac0b8b in sched_yield () from /lib64/libc.so.6
#5  0x0000151acdbd73dd in __gthread_yield () at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/el8_amd64_gcc12/external/gcc/12.3.1-40d504be6370b5a30e3947a6e575ca28/include/c++/12.3.1/x86_64-redhat-linux-gnu/bits/gthr-default.h:693
#6  std::this_thread::yield () at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/el8_amd64_gcc12/external/gcc/12.3.1-40d504be6370b5a30e3947a6e575ca28/include/c++/12.3.1/bits/std_thread.h:353
#7  tbb::detail::d0::atomic_backoff::pause (this=<synthetic pointer>) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-e785b749a0b6cb9c66dc1d78066210e0/tbb-v2021.9.0/src/tbb/../../include/oneapi/tbb/detail/_utils.h:71
#8  tbb::detail::r1::task_stream<(tbb::detail::r1::task_stream_accessor_type)0>::pop<tbb::detail::r1::subsequent_lane_selector> (next_lane=..., this=<optimized out>) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-e785b749a0b6cb9c66dc1d78066210e0/tbb-v2021.9.0/src/tbb/task_stream.h:196
#9  tbb::detail::r1::arena::get_stream_task<(tbb::detail::r1::task_stream_accessor_type)0> (this=<optimized out>, hint=@0x151aca5bb880: 4, stream=...) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-e785b749a0b6cb9c66dc1d78066210e0/tbb-v2021.9.0/src/tbb/arena.h:601
#10 tbb::detail::r1::task_dispatcher::get_stream_or_critical_task(tbb::detail::r1::execution_data_ext&, tbb::detail::r1::arena&, tbb::detail::r1::task_stream<(tbb::detail::r1::task_stream_accessor_type)0>&, unsigned int&, long, bool) [clone .isra.0] (this=this@entry=0x151aca5bbc80, ed=..., stream=..., hint=@0x151aca5bb880: 4, isolation=isolation@entry=0, critical_allowed=critical_allowed@entry=true, a=...) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-e785b749a0b6cb9c66dc1d78066210e0/tbb-v2021.9.0/src/tbb/task_dispatcher.h:154
#11 0x0000151acdbc5dbe in tbb::detail::r1::task_dispatcher::receive_or_steal_task<true, tbb::detail::r1::outermost_worker_waiter> (this=<optimized out>, tls=..., ed=..., waiter=..., isolation=<optimized out>, fifo_allowed=<optimized out>, critical_allowed=<optimized out>) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-e785b749a0b6cb9c66dc1d78066210e0/tbb-v2021.9.0/src/tbb/task_dispatcher.h:213
#12 0x0000151acdbd1cb2 in tbb::detail::r1::task_dispatcher::local_wait_for_all<false, tbb::detail::r1::outermost_worker_waiter> (t=0x0, waiter=..., this=0x151aca5bbc80) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-e785b749a0b6cb9c66dc1d78066210e0/tbb-v2021.9.0/src/tbb/task_dispatcher.h:350
#13 tbb::detail::r1::task_dispatcher::local_wait_for_all<tbb::detail::r1::outermost_worker_waiter> (t=0x0, waiter=..., this=0x151aca5bbc80) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-e785b749a0b6cb9c66dc1d78066210e0/tbb-v2021.9.0/src/tbb/task_dispatcher.h:458
#14 tbb::detail::r1::arena::process (tls=..., this=<optimized out>) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-e785b749a0b6cb9c66dc1d78066210e0/tbb-v2021.9.0/src/tbb/arena.cpp:137
#15 tbb::detail::r1::market::process (this=<optimized out>, j=...) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-e785b749a0b6cb9c66dc1d78066210e0/tbb-v2021.9.0/src/tbb/market.cpp:599
#16 0x0000151acdbd3cee in tbb::detail::r1::rml::private_worker::run (this=0x151ac4aaee00) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-e785b749a0b6cb9c66dc1d78066210e0/tbb-v2021.9.0/src/tbb/private_server.cpp:271
#17 tbb::detail::r1::rml::private_worker::thread_routine (arg=0x151ac4aaee00) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-e785b749a0b6cb9c66dc1d78066210e0/tbb-v2021.9.0/src/tbb/private_server.cpp:221
#18 0x0000151acbe661ca in start_thread () from /lib64/libpthread.so.0
#19 0x0000151acbac18d3 in clone () from /lib64/libc.so.6

Thread 8 (Thread 0x151a737fc700 (LWP 303307) "cmsRun"):
#0  0x0000151acbb90098 in nanosleep () from /lib64/libc.so.6
#1  0x0000151acbb8ff9e in sleep () from /lib64/libc.so.6
#2  0x0000151ac45b9360 in sig_pause_for_stacktrace () from /data/cmsbld/jenkins/workspace/ib-run-pr-tests/CMSSW_15_0_X_2024-11-25-1100/lib/el8_amd64_gcc12/pluginFWCoreServicesPlugins.so
#3  <signal handler called>
#4  0x0000151acbac141d in syscall () from /lib64/libc.so.6
#5  0x0000151acdbd3fd2 in tbb::detail::r1::futex_wait (comparand=2, futex=0x151ac4aaef24) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-e785b749a0b6cb9c66dc1d78066210e0/tbb-v2021.9.0/src/tbb/semaphore.h:100
#6  tbb::detail::r1::binary_semaphore::P (this=0x151ac4aaef24) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-e785b749a0b6cb9c66dc1d78066210e0/tbb-v2021.9.0/src/tbb/semaphore.h:253
#7  tbb::detail::r1::rml::internal::thread_monitor::wait (this=0x151ac4aaef20) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-e785b749a0b6cb9c66dc1d78066210e0/tbb-v2021.9.0/src/tbb/rml_thread_monitor.h:235
#8  tbb::detail::r1::rml::private_worker::run (this=0x151ac4aaef00) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-e785b749a0b6cb9c66dc1d78066210e0/tbb-v2021.9.0/src/tbb/private_server.cpp:273
#9  tbb::detail::r1::rml::private_worker::thread_routine (arg=0x151ac4aaef00) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-e785b749a0b6cb9c66dc1d78066210e0/tbb-v2021.9.0/src/tbb/private_server.cpp:221
#10 0x0000151acbe661ca in start_thread () from /lib64/libpthread.so.0
#11 0x0000151acbac18d3 in clone () from /lib64/libc.so.6

Thread 7 (Thread 0x151a741fd700 (LWP 303306) "cmsRun"):
#0  0x0000151acbb90098 in nanosleep () from /lib64/libc.so.6
#1  0x0000151acbb8ff9e in sleep () from /lib64/libc.so.6
#2  0x0000151ac45b9360 in sig_pause_for_stacktrace () from /data/cmsbld/jenkins/workspace/ib-run-pr-tests/CMSSW_15_0_X_2024-11-25-1100/lib/el8_amd64_gcc12/pluginFWCoreServicesPlugins.so
#3  <signal handler called>
#4  0x0000151acbac141d in syscall () from /lib64/libc.so.6
#5  0x0000151acdbd3fd2 in tbb::detail::r1::futex_wait (comparand=2, futex=0x151ac4aaf024) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-e785b749a0b6cb9c66dc1d78066210e0/tbb-v2021.9.0/src/tbb/semaphore.h:100
#6  tbb::detail::r1::binary_semaphore::P (this=0x151ac4aaf024) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-e785b749a0b6cb9c66dc1d78066210e0/tbb-v2021.9.0/src/tbb/semaphore.h:253
#7  tbb::detail::r1::rml::internal::thread_monitor::wait (this=0x151ac4aaf020) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-e785b749a0b6cb9c66dc1d78066210e0/tbb-v2021.9.0/src/tbb/rml_thread_monitor.h:235
#8  tbb::detail::r1::rml::private_worker::run (this=0x151ac4aaf000) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-e785b749a0b6cb9c66dc1d78066210e0/tbb-v2021.9.0/src/tbb/private_server.cpp:273
#9  tbb::detail::r1::rml::private_worker::thread_routine (arg=0x151ac4aaf000) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-e785b749a0b6cb9c66dc1d78066210e0/tbb-v2021.9.0/src/tbb/private_server.cpp:221
#10 0x0000151acbe661ca in start_thread () from /lib64/libpthread.so.0
#11 0x0000151acbac18d3 in clone () from /lib64/libc.so.6

Thread 6 (Thread 0x151a74bfe700 (LWP 303305) "cmsRun"):
#0  0x0000151acbb90098 in nanosleep () from /lib64/libc.so.6
#1  0x0000151acbb8ff9e in sleep () from /lib64/libc.so.6
#2  0x0000151ac45b9360 in sig_pause_for_stacktrace () from /data/cmsbld/jenkins/workspace/ib-run-pr-tests/CMSSW_15_0_X_2024-11-25-1100/lib/el8_amd64_gcc12/pluginFWCoreServicesPlugins.so
#3  <signal handler called>
#4  0x0000151acbac141d in syscall () from /lib64/libc.so.6
#5  0x0000151acdbd3fd2 in tbb::detail::r1::futex_wait (comparand=2, futex=0x151ac4aaeea4) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-e785b749a0b6cb9c66dc1d78066210e0/tbb-v2021.9.0/src/tbb/semaphore.h:100
#6  tbb::detail::r1::binary_semaphore::P (this=0x151ac4aaeea4) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-e785b749a0b6cb9c66dc1d78066210e0/tbb-v2021.9.0/src/tbb/semaphore.h:253
#7  tbb::detail::r1::rml::internal::thread_monitor::wait (this=0x151ac4aaeea0) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-e785b749a0b6cb9c66dc1d78066210e0/tbb-v2021.9.0/src/tbb/rml_thread_monitor.h:235
#8  tbb::detail::r1::rml::private_worker::run (this=0x151ac4aaee80) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-e785b749a0b6cb9c66dc1d78066210e0/tbb-v2021.9.0/src/tbb/private_server.cpp:273
#9  tbb::detail::r1::rml::private_worker::thread_routine (arg=0x151ac4aaee80) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-e785b749a0b6cb9c66dc1d78066210e0/tbb-v2021.9.0/src/tbb/private_server.cpp:221
#10 0x0000151acbe661ca in start_thread () from /lib64/libpthread.so.0
#11 0x0000151acbac18d3 in clone () from /lib64/libc.so.6

Thread 5 (Thread 0x151a757ff700 (LWP 303304) "cmsRun"):
#0  0x0000151acbb90098 in nanosleep () from /lib64/libc.so.6
#1  0x0000151acbb8ff9e in sleep () from /lib64/libc.so.6
#2  0x0000151ac45b9360 in sig_pause_for_stacktrace () from /data/cmsbld/jenkins/workspace/ib-run-pr-tests/CMSSW_15_0_X_2024-11-25-1100/lib/el8_amd64_gcc12/pluginFWCoreServicesPlugins.so
#3  <signal handler called>
#4  0x0000151acbac141d in syscall () from /lib64/libc.so.6
#5  0x0000151acdbd3fd2 in tbb::detail::r1::futex_wait (comparand=2, futex=0x151ac4aaefa4) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-e785b749a0b6cb9c66dc1d78066210e0/tbb-v2021.9.0/src/tbb/semaphore.h:100
#6  tbb::detail::r1::binary_semaphore::P (this=0x151ac4aaefa4) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-e785b749a0b6cb9c66dc1d78066210e0/tbb-v2021.9.0/src/tbb/semaphore.h:253
#7  tbb::detail::r1::rml::internal::thread_monitor::wait (this=0x151ac4aaefa0) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-e785b749a0b6cb9c66dc1d78066210e0/tbb-v2021.9.0/src/tbb/rml_thread_monitor.h:235
#8  tbb::detail::r1::rml::private_worker::run (this=0x151ac4aaef80) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-e785b749a0b6cb9c66dc1d78066210e0/tbb-v2021.9.0/src/tbb/private_server.cpp:273
#9  tbb::detail::r1::rml::private_worker::thread_routine (arg=0x151ac4aaef80) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-e785b749a0b6cb9c66dc1d78066210e0/tbb-v2021.9.0/src/tbb/private_server.cpp:221
#10 0x0000151acbe661ca in start_thread () from /lib64/libpthread.so.0
#11 0x0000151acbac18d3 in clone () from /lib64/libc.so.6

Thread 4 (Thread 0x151a769f4700 (LWP 303303) "cmsRun"):
#0  0x0000151acbb90098 in nanosleep () from /lib64/libc.so.6
#1  0x0000151acbb8ff9e in sleep () from /lib64/libc.so.6
#2  0x0000151ac45b9360 in sig_pause_for_stacktrace () from /data/cmsbld/jenkins/workspace/ib-run-pr-tests/CMSSW_15_0_X_2024-11-25-1100/lib/el8_amd64_gcc12/pluginFWCoreServicesPlugins.so
#3  <signal handler called>
#4  0x0000151acbac141d in syscall () from /lib64/libc.so.6
#5  0x0000151acdbd3fd2 in tbb::detail::r1::futex_wait (comparand=2, futex=0x151ac4aaf124) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-e785b749a0b6cb9c66dc1d78066210e0/tbb-v2021.9.0/src/tbb/semaphore.h:100
#6  tbb::detail::r1::binary_semaphore::P (this=0x151ac4aaf124) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-e785b749a0b6cb9c66dc1d78066210e0/tbb-v2021.9.0/src/tbb/semaphore.h:253
#7  tbb::detail::r1::rml::internal::thread_monitor::wait (this=0x151ac4aaf120) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-e785b749a0b6cb9c66dc1d78066210e0/tbb-v2021.9.0/src/tbb/rml_thread_monitor.h:235
#8  tbb::detail::r1::rml::private_worker::run (this=0x151ac4aaf100) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-e785b749a0b6cb9c66dc1d78066210e0/tbb-v2021.9.0/src/tbb/private_server.cpp:273
#9  tbb::detail::r1::rml::private_worker::thread_routine (arg=0x151ac4aaf100) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-e785b749a0b6cb9c66dc1d78066210e0/tbb-v2021.9.0/src/tbb/private_server.cpp:221
#10 0x0000151acbe661ca in start_thread () from /lib64/libpthread.so.0
#11 0x0000151acbac18d3 in clone () from /lib64/libc.so.6

Thread 3 (Thread 0x151a773f5700 (LWP 303302) "cmsRun"):
#0  0x0000151acbb90098 in nanosleep () from /lib64/libc.so.6
#1  0x0000151acbb8ff9e in sleep () from /lib64/libc.so.6
#2  0x0000151ac45b9360 in sig_pause_for_stacktrace () from /data/cmsbld/jenkins/workspace/ib-run-pr-tests/CMSSW_15_0_X_2024-11-25-1100/lib/el8_amd64_gcc12/pluginFWCoreServicesPlugins.so
#3  <signal handler called>
#4  0x0000151acbac141d in syscall () from /lib64/libc.so.6
#5  0x0000151acdbd3fd2 in tbb::detail::r1::futex_wait (comparand=2, futex=0x151ac4aaf0a4) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-e785b749a0b6cb9c66dc1d78066210e0/tbb-v2021.9.0/src/tbb/semaphore.h:100
#6  tbb::detail::r1::binary_semaphore::P (this=0x151ac4aaf0a4) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-e785b749a0b6cb9c66dc1d78066210e0/tbb-v2021.9.0/src/tbb/semaphore.h:253
#7  tbb::detail::r1::rml::internal::thread_monitor::wait (this=0x151ac4aaf0a0) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-e785b749a0b6cb9c66dc1d78066210e0/tbb-v2021.9.0/src/tbb/rml_thread_monitor.h:235
#8  tbb::detail::r1::rml::private_worker::run (this=0x151ac4aaf080) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-e785b749a0b6cb9c66dc1d78066210e0/tbb-v2021.9.0/src/tbb/private_server.cpp:273
#9  tbb::detail::r1::rml::private_worker::thread_routine (arg=0x151ac4aaf080) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-e785b749a0b6cb9c66dc1d78066210e0/tbb-v2021.9.0/src/tbb/private_server.cpp:221
#10 0x0000151acbe661ca in start_thread () from /lib64/libpthread.so.0
#11 0x0000151acbac18d3 in clone () from /lib64/libc.so.6

Thread 2 (Thread 0x151aa0303700 (LWP 303109) "cmsRun"):
#0  0x0000151acbe706a2 in waitpid () from /lib64/libpthread.so.0
#1  0x0000151ac45b94b7 in edm::service::cmssw_stacktrace_fork() () from /data/cmsbld/jenkins/workspace/ib-run-pr-tests/CMSSW_15_0_X_2024-11-25-1100/lib/el8_amd64_gcc12/pluginFWCoreServicesPlugins.so
#2  0x0000151ac45bc2ea in edm::service::InitRootHandlers::stacktraceHelperThread() () from /data/cmsbld/jenkins/workspace/ib-run-pr-tests/CMSSW_15_0_X_2024-11-25-1100/lib/el8_amd64_gcc12/pluginFWCoreServicesPlugins.so
#3  0x0000151acc4d8a73 in std::execute_native_thread_routine (__p=0x151aa4524ee0) at ../../../../../libstdc++-v3/src/c++11/thread.cc:82
#4  0x0000151acbe661ca in start_thread () from /lib64/libpthread.so.0
#5  0x0000151acbac18d3 in clone () from /lib64/libc.so.6

Thread 1 (Thread 0x151acdafe680 (LWP 302301) "cmsRun"):
#0  0x0000151acbbbaac1 in poll () from /lib64/libc.so.6
#1  0x0000151ac45bc507 in edm::service::InitRootHandlers::stacktraceFromThread() () from /data/cmsbld/jenkins/workspace/ib-run-pr-tests/CMSSW_15_0_X_2024-11-25-1100/lib/el8_amd64_gcc12/pluginFWCoreServicesPlugins.so
#2  0x0000151ac45bc704 in sig_dostack_then_abort () from /data/cmsbld/jenkins/workspace/ib-run-pr-tests/CMSSW_15_0_X_2024-11-25-1100/lib/el8_amd64_gcc12/pluginFWCoreServicesPlugins.so
#3  <signal handler called>
#4  tbb::detail::r1::spawn (t=..., ctx=...) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-e785b749a0b6cb9c66dc1d78066210e0/tbb-v2021.9.0/src/tbb/task_dispatcher.cpp:36
#5  0x0000151ace72d298 in tbb::detail::d1::function_task<edm::WaitingTaskHolder::doneWaiting(std::__exception_ptr::exception_ptr)::{lambda()#1}>::execute(tbb::detail::d1::execution_data&) () from /data/cmsbld/jenkins/workspace/ib-run-pr-tests/CMSSW_15_0_X_2024-11-25-1100/lib/el8_amd64_gcc12/libFWCoreFramework.so
#6  0x0000151acdbda3e1 in tbb::detail::r1::task_dispatcher::local_wait_for_all<false, tbb::detail::r1::external_waiter> (waiter=..., t=<optimized out>, this=0x151aca5bb900) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-e785b749a0b6cb9c66dc1d78066210e0/tbb-v2021.9.0/src/tbb/task_dispatcher.h:322
#7  tbb::detail::r1::task_dispatcher::local_wait_for_all<tbb::detail::r1::external_waiter> (waiter=..., t=<optimized out>, this=0x151aca5bb900) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-e785b749a0b6cb9c66dc1d78066210e0/tbb-v2021.9.0/src/tbb/task_dispatcher.h:458
#8  tbb::detail::r1::task_dispatcher::execute_and_wait (t=<optimized out>, wait_ctx=..., w_ctx=...) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-e785b749a0b6cb9c66dc1d78066210e0/tbb-v2021.9.0/src/tbb/task_dispatcher.cpp:168
#9  0x0000151ace742d1b in edm::FinalWaitingTask::wait() () from /data/cmsbld/jenkins/workspace/ib-run-pr-tests/CMSSW_15_0_X_2024-11-25-1100/lib/el8_amd64_gcc12/libFWCoreFramework.so
#10 0x0000151ace74f986 in edm::EventProcessor::endUnfinishedRun(bool) () from /data/cmsbld/jenkins/workspace/ib-run-pr-tests/CMSSW_15_0_X_2024-11-25-1100/lib/el8_amd64_gcc12/libFWCoreFramework.so
#11 0x0000151ace750cdd in edm::EventProcessor::runToCompletion() () from /data/cmsbld/jenkins/workspace/ib-run-pr-tests/CMSSW_15_0_X_2024-11-25-1100/lib/el8_amd64_gcc12/libFWCoreFramework.so
#12 0x000000000040840c in tbb::detail::d1::task_arena_function<main::{lambda()#1}::operator()() const::{lambda()#1}, void>::operator()() const ()
#13 0x0000151acdbc69ad in tbb::detail::r1::task_arena_impl::execute (ta=..., d=...) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-e785b749a0b6cb9c66dc1d78066210e0/tbb-v2021.9.0/src/tbb/arena.cpp:688
#14 0x000000000040a0f2 in main::{lambda()#1}::operator()() const ()
#15 0x0000000000405100 in main ()

Current Modules:

Module: none 

@Dr15Jones
Copy link
Contributor

Is it possible an exception went off and we aren't properly catching it?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

7 participants