Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[BUG]: Unit test TestControlPlane.LifeCycle fails with coredump #440

Open
2 tasks done
yczhang-nv opened this issue Feb 9, 2024 · 1 comment
Open
2 tasks done

[BUG]: Unit test TestControlPlane.LifeCycle fails with coredump #440

yczhang-nv opened this issue Feb 9, 2024 · 1 comment
Labels
bug Something isn't working

Comments

@yczhang-nv
Copy link

yczhang-nv commented Feb 9, 2024 •
edited

Version

24.03

Which installation method(s) does this occur on?

Docker

Describe the bug.

Unit test TestControlPlane.LifeCycle fails with coredump.

Minimum reproducible example

(mrc) root@2c839dbc5f33:/work# GLOG_v=5  ./build/cpp/mrc/src/tests/test_mrc_private.x --gtest_filter="TestControlPlane.LifeCycle"

Relevant log output

Note: Google Test filter = TestControlPlane.LifeCycle
[==========] Running 1 test from 1 test suite.
[----------] Global test environment set-up.
[----------] 1 test from TestControlPlane
[ RUN      ] TestControlPlane.LifeCycle
I20240209 00:12:38.714006  5677 topology.cpp:167] dropping gpu: [Tesla V100-SXM2-32GB; 32.0 GiB; cpu_set: 20-39,60-79; pcie_bus_id: 00000000:85:00.0; cuda_device_id: 4] because restrict_gpus is set to true; fails to overlap with topo cpu_set 0
I20240209 00:12:38.714409  5677 topology.cpp:167] dropping gpu: [Tesla V100-SXM2-32GB; 32.0 GiB; cpu_set: 20-39,60-79; pcie_bus_id: 00000000:86:00.0; cuda_device_id: 5] because restrict_gpus is set to true; fails to overlap with topo cpu_set 0
I20240209 00:12:38.714455  5677 topology.cpp:167] dropping gpu: [Tesla V100-SXM2-32GB; 32.0 GiB; cpu_set: 20-39,60-79; pcie_bus_id: 00000000:89:00.0; cuda_device_id: 6] because restrict_gpus is set to true; fails to overlap with topo cpu_set 0
I20240209 00:12:38.714490  5677 topology.cpp:167] dropping gpu: [Tesla V100-SXM2-32GB; 32.0 GiB; cpu_set: 20-39,60-79; pcie_bus_id: 00000000:8A:00.0; cuda_device_id: 7] because restrict_gpus is set to true; fails to overlap with topo cpu_set 0
I20240209 00:12:38.714525  5677 topology.cpp:215] topology restricted to cpu_set: 0
F20240209 00:12:38.714602  5677 partitions.cpp:125] fatal: cpu_count=1 is less than the number of cuda devices; unable to allocate 1 cpu cores per device
*** Check failure stack trace: ***
    @     0x7fbcbddc7f8d  google::LogMessage::Fail()
    @     0x7fbcbddcbe67  google::LogMessage::SendToLog()
    @     0x7fbcbddc7a55  google::LogMessage::Flush()
    @     0x7fbcbddc8eaa  google::LogMessageFatal::~LogMessageFatal()
    @     0x7fbcbe4ed54b  mrc::system::Partitions::Partitions()
    @     0x7fbcbe4ec38e  mrc::system::Partitions::Partitions()
    @     0x7fbcbe4ff6fc  _ZSt12construct_atIN3mrc6system10PartitionsEJRNS1_16SystemDefinitionEEEDTgsnwcvPvLi0E_T_pispcl7declvalIT0_EEEEPS6_DpOS7_
    @     0x7fbcbe4ff758  std::allocator_traits<>::construct<>()
    @     0x7fbcbe4ff1a2  std::_Sp_counted_ptr_inplace<>::_Sp_counted_ptr_inplace<>()
    @     0x7fbcbe4feaa5  std::__shared_count<>::__shared_count<>()
    @     0x7fbcbe4fe5a0  std::__shared_ptr<>::__shared_ptr<>()
    @     0x7fbcbe4fdd1d  std::shared_ptr<>::shared_ptr<>()
    @     0x7fbcbe4fd495  std::allocate_shared<>()
    @     0x7fbcbe4fcebd  std::make_shared<>()
    @     0x7fbcbe4fc65d  mrc::system::SystemDefinition::SystemDefinition()
    @     0x7fbcbe4fc70f  mrc::system::SystemDefinition::SystemDefinition()
    @     0x55c69ed188c8  std::make_unique<>()
    @     0x55c69ed1841d  mrc::tests::make_system()
    @     0x55c69ed9df87  make_runtime()
    @     0x55c69ed9e1fd  TestControlPlane_LifeCycle_Test::TestBody()
    @     0x7fbcbe02540e  testing::internal::HandleExceptionsInMethodIfSupported<>()
    @     0x7fbcbe0256a1  testing::Test::Run()
    @     0x7fbcbe025a2f  testing::TestInfo::Run()
    @     0x7fbcbe025eff  testing::TestSuite::Run()
    @     0x7fbcbe031423  testing::internal::UnitTestImpl::RunAllTests()
    @     0x7fbcbe025fdd  testing::UnitTest::Run()
    @     0x55c69ee58027  RUN_ALL_TESTS()
    @     0x55c69ee580d1  main
    @     0x7fbcbca1bd90  (unknown)
    @     0x7fbcbca1be40  __libc_start_main
    @     0x55c69ed18079  (unknown)
    @              (nil)  (unknown)
Aborted (core dumped)

Full env printout

<details><summary>Click here to see environment details</summary><pre>
     
     **git***
     commit cf3d20fc6d09d756b8cfc338c222d5946fba370a (HEAD -> yuchen-work, upstream/branch-24.03, origin/yuchen-work, origin/branch-24.03, origin/HEAD, branch-24.03)
     Author: Christopher Harris <[email protected]>
     Date:   Wed Feb 7 15:37:32 2024 -0600
     
     Update Conda channels to prioritize `conda-forge` over `nvidia` (#436)
     
     Closes https://github.com/nv-morpheus/MRC/issues/435
     Closes https://github.com/nv-morpheus/MRC/issues/424
     Closes https://github.com/nv-morpheus/MRC/issues/423
     Closes https://github.com/nv-morpheus/MRC/issues/422
     
     Authors:
     - Christopher Harris (https://github.com/cwharris)
     
     Approvers:
     - Michael Demoret (https://github.com/mdemoret-nv)
     
     URL: https://github.com/nv-morpheus/MRC/pull/436
     **git submodules***
     eb55e1acb73df1dbf4c1b69f17c918c661921c3c external/utilities (v24.03.00a-5-geb55e1a)
     
     ***OS Information***
     DISTRIB_ID=Ubuntu
     DISTRIB_RELEASE=22.04
     DISTRIB_CODENAME=jammy
     DISTRIB_DESCRIPTION="Ubuntu 22.04.3 LTS"
     PRETTY_NAME="Ubuntu 22.04.3 LTS"
     NAME="Ubuntu"
     VERSION_ID="22.04"
     VERSION="22.04.3 LTS (Jammy Jellyfish)"
     VERSION_CODENAME=jammy
     ID=ubuntu
     ID_LIKE=debian
     HOME_URL="https://www.ubuntu.com/"
     SUPPORT_URL="https://help.ubuntu.com/"
     BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
     PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
     UBUNTU_CODENAME=jammy
     Linux 2c839dbc5f33 5.4.0-148-generic #165-Ubuntu SMP Tue Apr 18 08:53:12 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
     
     ***GPU Information***
     Fri Feb  9 00:19:39 2024
     +-----------------------------------------------------------------------------+
     | NVIDIA-SMI 525.105.17   Driver Version: 525.105.17   CUDA Version: 12.1     |
     |-------------------------------+----------------------+----------------------+
     | GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
     | Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
     |                               |                      |               MIG M. |
     |===============================+======================+======================|
     |   0  Tesla V100-SXM2...  On   | 00000000:06:00.0 Off |                    0 |
     | N/A   31C    P0    57W / 300W |  31531MiB / 32768MiB |      0%      Default |
     |                               |                      |                  N/A |
     +-------------------------------+----------------------+----------------------+
     |   1  Tesla V100-SXM2...  On   | 00000000:07:00.0 Off |                    0 |
     | N/A   32C    P0    43W / 300W |      5MiB / 32768MiB |      0%      Default |
     |                               |                      |                  N/A |
     +-------------------------------+----------------------+----------------------+
     |   2  Tesla V100-SXM2...  On   | 00000000:0A:00.0 Off |                    0 |
     | N/A   31C    P0    42W / 300W |      5MiB / 32768MiB |      0%      Default |
     |                               |                      |                  N/A |
     +-------------------------------+----------------------+----------------------+
     |   3  Tesla V100-SXM2...  On   | 00000000:0B:00.0 Off |                    0 |
     | N/A   31C    P0    56W / 300W |   2225MiB / 32768MiB |      0%      Default |
     |                               |                      |                  N/A |
     +-------------------------------+----------------------+----------------------+
     |   4  Tesla V100-SXM2...  On   | 00000000:85:00.0 Off |                    0 |
     | N/A   29C    P0    44W / 300W |      5MiB / 32768MiB |      0%      Default |
     |                               |                      |                  N/A |
     +-------------------------------+----------------------+----------------------+
     |   5  Tesla V100-SXM2...  On   | 00000000:86:00.0 Off |                    0 |
     | N/A   34C    P0    56W / 300W |   1161MiB / 32768MiB |      0%      Default |
     |                               |                      |                  N/A |
     +-------------------------------+----------------------+----------------------+
     |   6  Tesla V100-SXM2...  On   | 00000000:89:00.0 Off |                    0 |
     | N/A   34C    P0    57W / 300W |   5419MiB / 32768MiB |      0%      Default |
     |                               |                      |                  N/A |
     +-------------------------------+----------------------+----------------------+
     |   7  Tesla V100-SXM2...  On   | 00000000:8A:00.0 Off |                    0 |
     | N/A   30C    P0    42W / 300W |      5MiB / 32768MiB |      0%      Default |
     |                               |                      |                  N/A |
     +-------------------------------+----------------------+----------------------+
     
     +-----------------------------------------------------------------------------+
     | Processes:                                                                  |
     |  GPU   GI   CI        PID   Type   Process name                  GPU Memory |
     |        ID   ID                                                   Usage      |
     |=============================================================================|
     +-----------------------------------------------------------------------------+
     
     ***CPU***
     Architecture:                    x86_64
     CPU op-mode(s):                  32-bit, 64-bit
     Address sizes:                   46 bits physical, 48 bits virtual
     Byte Order:                      Little Endian
     CPU(s):                          80
     On-line CPU(s) list:             0-79
     Vendor ID:                       GenuineIntel
     Model name:                      Intel(R) Xeon(R) CPU E5-2698 v4 @ 2.20GHz
     CPU family:                      6
     Model:                           79
     Thread(s) per core:              2
     Core(s) per socket:              20
     Socket(s):                       2
     Stepping:                        1
     CPU max MHz:                     3600.0000
     CPU min MHz:                     1200.0000
     BogoMIPS:                        4389.97
     Flags:                           fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc cpuid aperfmperf pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid dca sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm 3dnowprefetch cpuid_fault epb cat_l3 cdp_l3 invpcid_single pti intel_ppin ssbd ibrs ibpb stibp tpr_shadow vnmi flexpriority ept vpid ept_ad fsgsbase tsc_adjust bmi1 hle avx2 smep bmi2 erms invpcid rtm cqm rdt_a rdseed adx smap intel_pt xsaveopt cqm_llc cqm_occup_llc cqm_mbm_total cqm_mbm_local dtherm ida arat pln pts md_clear flush_l1d
     Virtualization:                  VT-x
     L1d cache:                       1.3 MiB (40 instances)
     L1i cache:                       1.3 MiB (40 instances)
     L2 cache:                        10 MiB (40 instances)
     L3 cache:                        100 MiB (2 instances)
     NUMA node(s):                    2
     NUMA node0 CPU(s):               0-19,40-59
     NUMA node1 CPU(s):               20-39,60-79
     Vulnerability Itlb multihit:     KVM: Mitigation: Split huge pages
     Vulnerability L1tf:              Mitigation; PTE Inversion; VMX conditional cache flushes, SMT vulnerable
     Vulnerability Mds:               Mitigation; Clear CPU buffers; SMT vulnerable
     Vulnerability Meltdown:          Mitigation; PTI
     Vulnerability Mmio stale data:   Mitigation; Clear CPU buffers; SMT vulnerable
     Vulnerability Retbleed:          Not affected
     Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl and seccomp
     Vulnerability Spectre v1:        Mitigation; usercopy/swapgs barriers and __user pointer sanitization
     Vulnerability Spectre v2:        Mitigation; Retpolines, IBPB conditional, IBRS_FW, STIBP conditional, RSB filling, PBRSB-eIBRS Not affected
     Vulnerability Srbds:             Not affected
     Vulnerability Tsx async abort:   Mitigation; Clear CPU buffers; SMT vulnerable
     
     ***CMake***
     /opt/conda/envs/mrc/bin/cmake
     cmake version 3.27.9
     
     CMake suite maintained and supported by Kitware (kitware.com/cmake).
     
     ***g++***
     /opt/conda/envs/mrc/bin/g++
     g++ (conda-forge gcc 11.2.0-16) 11.2.0
     Copyright (C) 2021 Free Software Foundation, Inc.
     This is free software; see the source for copying conditions.  There is NO
     warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
     
     
     ***nvcc***
     /opt/conda/envs/mrc/bin/nvcc
     nvcc: NVIDIA (R) Cuda compiler driver
     Copyright (c) 2005-2023 NVIDIA Corporation
     Built on Mon_Apr__3_17:16:06_PDT_2023
     Cuda compilation tools, release 12.1, V12.1.105
     Build cuda_12.1.r12.1/compiler.32688072_0
     
     ***Python***
     /opt/conda/envs/mrc/bin/python
     Python 3.10.13
     
     ***Environment Variables***
     PATH                            : /opt/conda/envs/mrc/bin:/opt/conda/condabin:/opt/conda/bin:/usr/local/nvidia/bin:/usr/local/cuda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
     LD_LIBRARY_PATH                 : /usr/local/nvidia/lib:/usr/local/nvidia/lib64
     NUMBAPRO_NVVM                   :
     NUMBAPRO_LIBDEVICE              :
     CONDA_PREFIX                    : /opt/conda/envs/mrc
     PYTHON_PATH                     :
     
     ***conda packages***
     /opt/conda/condabin/conda
     # packages in environment at /opt/conda/envs/mrc:
     #
     # Name                    Version                   Build  Channel
     _libgcc_mutex             0.1                 conda_forge    conda-forge
     _openmp_mutex             4.5                       2_gnu    conda-forge
     _sysroot_linux-64_curr_repodata_hack 3                   h69a702a_13    conda-forge
     atk-1.0                   2.38.0               hd4edc92_1    conda-forge
     bash-completion           2.11                 ha770c72_1    conda-forge
     benchmark                 1.8.3                h59595ed_0    conda-forge
     binutils                  2.36.1               hdd6e379_2    conda-forge
     binutils_impl_linux-64    2.36.1               h193b22a_2    conda-forge
     binutils_linux-64         2.36                hf3e587d_10    conda-forge
     boost-cpp                 1.84.0               h44aadfe_0    conda-forge
     brotli-python             1.1.0           py310hc6cd4ac_1    conda-forge
     bzip2                     1.0.8                hd590300_5    conda-forge
     c-ares                    1.26.0               hd590300_0    conda-forge
     c-compiler                1.5.2                h0b41bf4_0    conda-forge
     ca-certificates           2024.2.2             hbcca054_0    conda-forge
     cairo                     1.18.0               h3faef2a_0    conda-forge
     ccache                    4.8.1                h1fcd64f_0    conda-forge
     certifi                   2024.2.2           pyhd8ed1ab_0    conda-forge
     cffi                      1.16.0          py310h2fee648_0    conda-forge
     cfgv                      3.3.1              pyhd8ed1ab_0    conda-forge
     charset-normalizer        3.3.2              pyhd8ed1ab_0    conda-forge
     clang                     16.0.6               hda56bd4_4    conda-forge
     clang-16                  16.0.6          default_hb11cfb5_4    conda-forge
     clang-format              16.0.6          default_hb11cfb5_4    conda-forge
     clang-format-16           16.0.6          default_hb11cfb5_4    conda-forge
     clang-tools               16.0.6          default_hb11cfb5_4    conda-forge
     clangdev                  16.0.6          default_hb11cfb5_4    conda-forge
     clangxx                   16.0.6          default_hb11cfb5_4    conda-forge
     cmake                     3.27.9               hcfe8598_0    conda-forge
     codecov                   2.1.13             pyhd8ed1ab_0    conda-forge
     colorama                  0.4.6              pyhd8ed1ab_0    conda-forge
     coverage                  7.4.1           py310h2372a71_0    conda-forge
     cuda-cccl                 12.1.109             ha770c72_0    conda-forge
     cuda-cccl-impl            2.0.1                ha770c72_1    conda-forge
     cuda-cccl_linux-64        12.1.109             ha770c72_0    conda-forge
     cuda-command-line-tools   12.1.1               ha770c72_0    conda-forge
     cuda-cudart               12.1.105             hd3aeb46_0    conda-forge
     cuda-cudart-dev           12.1.105             hd3aeb46_0    conda-forge
     cuda-cudart-dev_linux-64  12.1.105             h59595ed_0    conda-forge
     cuda-cudart-static        12.1.105             hd3aeb46_0    conda-forge
     cuda-cudart-static_linux-64 12.1.105             h59595ed_0    conda-forge
     cuda-cudart_linux-64      12.1.105             h59595ed_0    conda-forge
     cuda-cupti                12.1.105             h59595ed_0    conda-forge
     cuda-cupti-dev            12.1.105             h59595ed_0    conda-forge
     cuda-driver-dev           12.1.105             hd3aeb46_0    conda-forge
     cuda-driver-dev_linux-64  12.1.105             h59595ed_0    conda-forge
     cuda-gdb                  12.1.105             hd47b8d6_0    conda-forge
     cuda-libraries-dev        12.1.1               ha770c72_0    conda-forge
     cuda-nsight               12.1.105             ha770c72_0    conda-forge
     cuda-nvcc                 12.1.105             hcdd1206_1    conda-forge
     cuda-nvcc-dev_linux-64    12.1.105             ha770c72_0    conda-forge
     cuda-nvcc-impl            12.1.105             hd3aeb46_0    conda-forge
     cuda-nvcc-tools           12.1.105             hd3aeb46_0    conda-forge
     cuda-nvcc_linux-64        12.1.105             h8a487aa_1    conda-forge
     cuda-nvdisasm             12.1.105             h59595ed_0    conda-forge
     cuda-nvml-dev             12.1.105             h59595ed_0    conda-forge
     cuda-nvprof               12.1.105             h59595ed_0    conda-forge
     cuda-nvrtc                12.1.105             hd3aeb46_0    conda-forge
     cuda-nvrtc-dev            12.1.105             hd3aeb46_0    conda-forge
     cuda-nvtx                 12.1.105             h59595ed_0    conda-forge
     cuda-nvvp                 12.1.105             h59595ed_0    conda-forge
     cuda-opencl               12.1.105             h59595ed_0    conda-forge
     cuda-opencl-dev           12.1.105             h59595ed_0    conda-forge
     cuda-profiler-api         12.1.105             ha770c72_0    conda-forge
     cuda-sanitizer-api        12.1.105             h59595ed_0    conda-forge
     cuda-tools                12.1.1               ha770c72_0    conda-forge
     cuda-version              12.1                 h1d6eff3_2    conda-forge
     cuda-visual-tools         12.1.1               ha770c72_0    conda-forge
     cxx-compiler              1.5.2                hf52228f_0    conda-forge
     distlib                   0.3.8              pyhd8ed1ab_0    conda-forge
     distro                    1.9.0              pyhd8ed1ab_0    conda-forge
     doxygen                   1.10.0               h661eb56_0    conda-forge
     exceptiongroup            1.2.0              pyhd8ed1ab_2    conda-forge
     expat                     2.5.0                hcb278e6_1    conda-forge
     filelock                  3.13.1             pyhd8ed1ab_0    conda-forge
     flake8                    7.0.0              pyhd8ed1ab_0    conda-forge
     fmt                       10.2.1               h00ab1b0_0    conda-forge
     font-ttf-dejavu-sans-mono 2.37                 hab24e00_0    conda-forge
     font-ttf-inconsolata      3.000                h77eed37_0    conda-forge
     font-ttf-source-code-pro  2.038                h77eed37_0    conda-forge
     font-ttf-ubuntu           0.83                 h77eed37_1    conda-forge
     fontconfig                2.14.2               h14ed4e7_0    conda-forge
     fonts-conda-ecosystem     1                             0    conda-forge
     fonts-conda-forge         1                             0    conda-forge
     freetype                  2.12.1               h267a509_2    conda-forge
     fribidi                   1.0.10               h36c2ea0_0    conda-forge
     gcc                       11.2.0              h702ea55_10    conda-forge
     gcc_impl_linux-64         11.2.0              h82a94d6_16    conda-forge
     gcc_linux-64              11.2.0              h39a9532_10    conda-forge
     gcovr                     5.2                pyhd8ed1ab_0    conda-forge
     gdb                       12.1            py310hd73dadb_0    conda-forge
     gdk-pixbuf                2.42.10              h829c605_4    conda-forge
     gds-tools                 1.6.1.9              hd3aeb46_0    conda-forge
     gettext                   0.21.1               h27087fc_0    conda-forge
     gflags                    2.2.2             he1b5a44_1004    conda-forge
     giflib                    5.2.1                h0b41bf4_3    conda-forge
     glog                      0.6.0                h6f12383_0    conda-forge
     gmp                       6.3.0                h59595ed_0    conda-forge
     graphite2                 1.3.13            h58526e2_1001    conda-forge
     graphviz                  9.0.0                h78e8752_1    conda-forge
     gtest                     1.14.0               h00ab1b0_1    conda-forge
     gtk2                      2.24.33              h7f000aa_3    conda-forge
     gts                       0.7.6                h977cf35_4    conda-forge
     gxx                       11.2.0              h702ea55_10    conda-forge
     gxx_impl_linux-64         11.2.0              h82a94d6_16    conda-forge
     gxx_linux-64              11.2.0              hacbe6df_10    conda-forge
     harfbuzz                  8.3.0                h3d44ed6_0    conda-forge
     icu                       73.2                 h59595ed_0    conda-forge
     identify                  2.5.33             pyhd8ed1ab_0    conda-forge
     idna                      3.6                pyhd8ed1ab_0    conda-forge
     importlib-metadata        7.0.1              pyha770c72_0    conda-forge
     include-what-you-use      0.20                 h59595ed_0    conda-forge
     iniconfig                 2.0.0              pyhd8ed1ab_0    conda-forge
     jinja2                    3.1.3              pyhd8ed1ab_0    conda-forge
     kernel-headers_linux-64   3.10.0              h4a8ded7_13    conda-forge
     keyutils                  1.6.1                h166bdaf_0    conda-forge
     krb5                      1.21.2               h659d440_0    conda-forge
     ld_impl_linux-64          2.36.1               hea4e1c9_2    conda-forge
     lerc                      4.0.0                h27087fc_0    conda-forge
     libabseil                 20230802.1      cxx17_h59595ed_0    conda-forge
     libblas                   3.9.0           21_linux64_openblas    conda-forge
     libboost                  1.84.0               h6fcfa73_0    conda-forge
     libboost-devel            1.84.0               h00ab1b0_0    conda-forge
     libboost-headers          1.84.0               ha770c72_0    conda-forge
     libcblas                  3.9.0           21_linux64_openblas    conda-forge
     libclang                  16.0.6          default_hb11cfb5_4    conda-forge
     libclang-cpp              16.0.6          default_hb11cfb5_4    conda-forge
     libclang-cpp16            16.0.6          default_hb11cfb5_4    conda-forge
     libclang13                16.0.6          default_ha2b6cf4_4    conda-forge
     libcublas                 12.1.3.1             hd3aeb46_0    conda-forge
     libcublas-dev             12.1.3.1             hd3aeb46_0    conda-forge
     libcufft                  11.0.2.54            hd3aeb46_0    conda-forge
     libcufft-dev              11.0.2.54            hd3aeb46_0    conda-forge
     libcufile                 1.6.1.9              hd3aeb46_0    conda-forge
     libcufile-dev             1.6.1.9              hd3aeb46_0    conda-forge
     libcurand                 10.3.2.106           hd3aeb46_0    conda-forge
     libcurand-dev             10.3.2.106           hd3aeb46_0    conda-forge
     libcurl                   8.5.0                hca28451_0    conda-forge
     libcusolver               11.4.5.107           hd3aeb46_0    conda-forge
     libcusolver-dev           11.4.5.107           hd3aeb46_0    conda-forge
     libcusparse               12.1.0.106           hd3aeb46_0    conda-forge
     libcusparse-dev           12.1.0.106           hd3aeb46_0    conda-forge
     libdeflate                1.19                 hd590300_0    conda-forge
     libedit                   3.1.20191231         he28a2e2_2    conda-forge
     libev                     4.33                 hd590300_2    conda-forge
     libexpat                  2.5.0                hcb278e6_1    conda-forge
     libffi                    3.4.2                h7f98852_5    conda-forge
     libgcc-devel_linux-64     11.2.0              h0952999_16    conda-forge
     libgcc-ng                 13.2.0               h807b86a_5    conda-forge
     libgd                     2.3.3                h119a65a_9    conda-forge
     libgfortran-ng            13.2.0               h69a702a_5    conda-forge
     libgfortran5              13.2.0               ha4646dd_5    conda-forge
     libglib                   2.78.3               h783c2da_0    conda-forge
     libgomp                   13.2.0               h807b86a_5    conda-forge
     libgrpc                   1.59.3               hd6c4280_0    conda-forge
     libhiredis                1.0.2                h2cc385e_0    conda-forge
     libhwloc                  2.9.2           default_h554bfaf_1009    conda-forge
     libiconv                  1.17                 hd590300_2    conda-forge
     libjpeg-turbo             3.0.0                hd590300_1    conda-forge
     liblapack                 3.9.0           21_linux64_openblas    conda-forge
     libllvm16                 16.0.6               h5cf9203_2    conda-forge
     libnghttp2                1.58.0               h47da74e_1    conda-forge
     libnl                     3.9.0                hd590300_0    conda-forge
     libnpp                    12.1.0.40            hd3aeb46_0    conda-forge
     libnpp-dev                12.1.0.40            hd3aeb46_0    conda-forge
     libnsl                    2.0.1                hd590300_0    conda-forge
     libnuma                   2.0.16               h0b41bf4_1    conda-forge
     libnvjitlink              12.1.105             hd3aeb46_0    conda-forge
     libnvjitlink-dev          12.1.105             hd3aeb46_0    conda-forge
     libnvjpeg                 12.2.0.2             h59595ed_0    conda-forge
     libnvjpeg-dev             12.2.0.2             ha770c72_0    conda-forge
     libopenblas               0.3.26          pthreads_h413a1c8_0    conda-forge
     libpng                    1.6.42               h2797004_0    conda-forge
     libprotobuf               4.24.4               hf27288f_0    conda-forge
     libre2-11                 2023.06.02           h7a70373_0    conda-forge
     librmm                    24.02.00a37     cuda12_240208_gf32d35b4_37    rapidsai-nightly
     librsvg                   2.56.3               h98fae49_0    conda-forge
     libsanitizer              11.2.0              he4da1e4_16    conda-forge
     libsqlite                 3.44.2               h2797004_0    conda-forge
     libssh2                   1.11.0               h0841786_0    conda-forge
     libstdcxx-devel_linux-64  11.2.0              h0952999_16    conda-forge
     libstdcxx-ng              13.2.0               h7e041cc_5    conda-forge
     libtiff                   4.6.0                ha9c0a0a_2    conda-forge
     libuuid                   2.38.1               h0b41bf4_0    conda-forge
     libuv                     1.46.0               hd590300_0    conda-forge
     libwebp                   1.3.2                h658648e_1    conda-forge
     libwebp-base              1.3.2                hd590300_0    conda-forge
     libxcb                    1.15                 h0b41bf4_0    conda-forge
     libxcrypt                 4.4.36               hd590300_1    conda-forge
     libxml2                   2.11.6               h232c23b_0    conda-forge
     libxslt                   1.1.37               h0054252_1    conda-forge
     libzlib                   1.2.13               hd590300_5    conda-forge
     llvm-tools                16.0.6               h5cf9203_2    conda-forge
     llvmdev                   16.0.6               h5cf9203_2    conda-forge
     lxml                      4.9.3           py310h9b7343a_3    conda-forge
     markupsafe                2.1.5           py310h2372a71_0    conda-forge
     mccabe                    0.7.0              pyhd8ed1ab_0    conda-forge
     mrc                       24.3.0a0+13.gcf3d20fc.dirty          pypi_0    pypi
     ncurses                   6.4                  h59595ed_2    conda-forge
     ninja                     1.11.1               h924138e_0    conda-forge
     nlohmann_json             3.9.1                h9c3ff4c_1    conda-forge
     nodeenv                   1.8.0              pyhd8ed1ab_0    conda-forge
     nsight-compute            2023.1.1.4                    0    nvidia
     numactl-libs-cos7-x86_64  2.0.12            h9b0a68f_1105    conda-forge
     numpy                     1.24.4          py310ha4c1d20_0    conda-forge
     ocl-icd                   2.3.1                h7f98852_0    conda-forge
     openssl                   3.2.1                hd590300_0    conda-forge
     packaging                 23.2               pyhd8ed1ab_0    conda-forge
     pango                     1.50.14              ha41ecd1_2    conda-forge
     pcre2                     10.42                hcad00b1_0    conda-forge
     pip                       24.0               pyhd8ed1ab_0    conda-forge
     pixman                    0.43.2               h59595ed_0    conda-forge
     pkg-config                0.29.2            h36c2ea0_1008    conda-forge
     platformdirs              4.2.0              pyhd8ed1ab_0    conda-forge
     pluggy                    1.4.0              pyhd8ed1ab_0    conda-forge
     pre-commit                3.6.0              pyha770c72_0    conda-forge
     pthread-stubs             0.4               h36c2ea0_1001    conda-forge
     pybind11-stubgen          0.10.5             pyhd8ed1ab_0    conda-forge
     pycodestyle               2.11.1             pyhd8ed1ab_0    conda-forge
     pycparser                 2.21               pyhd8ed1ab_0    conda-forge
     pyflakes                  3.2.0              pyhd8ed1ab_0    conda-forge
     pygments                  2.17.2             pyhd8ed1ab_0    conda-forge
     pysocks                   1.7.1              pyha2e5f31_6    conda-forge
     pytest                    7.4.4              pyhd8ed1ab_0    conda-forge
     pytest-asyncio            0.23.4             pyhd8ed1ab_0    conda-forge
     pytest-timeout            2.2.0              pyhd8ed1ab_0    conda-forge
     python                    3.10.13         hd12c33a_1_cpython    conda-forge
     python-graphviz           0.20.1             pyh22cad53_0    conda-forge
     python_abi                3.10                    4_cp310    conda-forge
     pyyaml                    6.0.1           py310h2372a71_1    conda-forge
     rdma-core                 50.0                 hd3aeb46_0    conda-forge
     re2                       2023.06.02           h2873b5e_0    conda-forge
     readline                  8.2                  h8228510_1    conda-forge
     requests                  2.31.0             pyhd8ed1ab_0    conda-forge
     rhash                     1.4.4                hd590300_0    conda-forge
     scikit-build              0.17.6             pyh4af843d_0    conda-forge
     setuptools                69.0.3             pyhd8ed1ab_0    conda-forge
     six                       1.16.0             pyh6c4a22f_0    conda-forge
     spdlog                    1.12.0               hd2e6256_2    conda-forge
     sysroot_linux-64          2.17                h4a8ded7_13    conda-forge
     tk                        8.6.13          noxft_h4845f30_101    conda-forge
     tomli                     2.0.1              pyhd8ed1ab_0    conda-forge
     typing-extensions         4.9.0                hd8ed1ab_0    conda-forge
     typing_extensions         4.9.0              pyha770c72_0    conda-forge
     tzdata                    2024a                h0c530f3_0    conda-forge
     ucx                       1.15.0               h6d2d1ec_3    conda-forge
     ukkonen                   1.0.1           py310hd41b1e2_4    conda-forge
     urllib3                   2.2.0              pyhd8ed1ab_0    conda-forge
     virtualenv                20.25.0            pyhd8ed1ab_0    conda-forge
     wheel                     0.42.0             pyhd8ed1ab_0    conda-forge
     xorg-kbproto              1.0.7             h7f98852_1002    conda-forge
     xorg-libice               1.1.1                hd590300_0    conda-forge
     xorg-libsm                1.2.4                h7391055_0    conda-forge
     xorg-libx11               1.8.7                h8ee46fc_0    conda-forge
     xorg-libxau               1.0.11               hd590300_0    conda-forge
     xorg-libxdmcp             1.1.3                h7f98852_0    conda-forge
     xorg-libxext              1.3.4                h0b41bf4_2    conda-forge
     xorg-libxrender           0.9.11               hd590300_0    conda-forge
     xorg-renderproto          0.11.1            h7f98852_1002    conda-forge
     xorg-xextproto            7.3.0             h0b41bf4_1003    conda-forge
     xorg-xproto               7.0.31            h7f98852_1007    conda-forge
     xz                        5.2.6                h166bdaf_0    conda-forge
     yaml                      0.2.5                h7f98852_2    conda-forge
     yapf                      0.40.1             pyhd8ed1ab_0    conda-forge
     zipp                      3.17.0             pyhd8ed1ab_0    conda-forge
     zlib                      1.2.13               hd590300_5    conda-forge
     zstd                      1.5.5                hfc55251_0    conda-forge
     
</pre></details>

Other/Misc.

No response

Code of Conduct

  • I agree to follow MRC's Code of Conduct
  • I have searched the open bugs and have found no duplicates for this bug report
@yczhang-nv yczhang-nv added the bug Something isn't working label Feb 9, 2024
@jarmak-nv jarmak-nv added the Needs Triage Requires attention label Feb 9, 2024
@jarmak-nv
Copy link
Contributor

Hi @yuchenz427!

Thanks for submitting this issue - our team has been notified and we'll get back to you as soon as we can!
In the meantime, feel free to add any relevant information to this issue.

@mdemoret-nv mdemoret-nv changed the title [BUG]: Unit test fails with coredump when running on DGX04 [BUG]: Unit test TestControlPlane.LifeCycle fails with coredump Feb 9, 2024
@yczhang-nv yczhang-nv removed the Needs Triage Requires attention label Aug 16, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug Something isn't working
Projects
Morpheus Boards
Status: Todo
Milestone
No milestone
Development

No branches or pull requests
2 participants
@jarmak-nv @yczhang-nv