Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Occasional segfaults on m1 apple silicon #203

Open
ericphanson opened this issue Apr 6, 2024 · 2 comments
Open

Occasional segfaults on m1 apple silicon #203

ericphanson opened this issue Apr 6, 2024 · 2 comments

Comments

@ericphanson
Copy link

ericphanson commented Apr 6, 2024 •
edited
Loading

I was able to repeat it once with this script, but it fails to consistently reproduce:

using StableRNGs, MLJ, MLJXGBoostInterface, DataFrames
X = float.(rand(StableRNG(1), [0,0,0,0,0,1], 10000, 4));
y = rand(StableRNG(2), 0:1, 10000);
classifier = Pipeline(; standardizer=Standardizer(), classifier=XGBoostClassifier());
mach = machine(classifier, DataFrame(X, :auto), coerce(y, OrderedFactor))
fit!(mach)

Here's the time that it happened:

julia> using StableRNGs, MLJ, MLJXGBoostInterface, DataFrames

julia> X = float.(rand(StableRNG(1), [0,0,0,0,1], 10000, 4))
10000×4 Matrix{Float64}:
 0.0  1.0  0.0  1.0
 0.0  1.0  0.0  0.0
 1.0  1.0  0.0  1.0
 0.0  0.0  0.0  0.0
 0.0  1.0  0.0  0.0
 0.0  0.0  0.0  0.0
 ⋮              
 0.0  0.0  0.0  0.0
 0.0  0.0  0.0  0.0
 0.0  1.0  0.0  0.0
 0.0  1.0  0.0  0.0
 0.0  0.0  0.0  0.0

julia> y = rand(StableRNG(2), 0:1, 10000);

julia> classifier = Pipeline(Standardizer(), XGBoostClassifier());

julia> mach = machine(classifier, DataFrame(X, :auto), coerce(y, OrderedFactor))
untrained Machine; does not cache data
  model: ProbabilisticPipeline(standardizer = Standardizer(features = Symbol[], …), …)
  args: 
    1:  Source @134 ⏎ Table{AbstractVector{Continuous}}
    2:  Source @800 ⏎ AbstractVector{OrderedFactor{2}}


julia> fit!(mach)
[ Info: Training machine(ProbabilisticPipeline(standardizer = Standardizer(features = Symbol[], …), …), …).
[ Info: Training machine(:standardizer, …).
[ Info: Training machine(:xg_boost_classifier, …).
[ Info: XGBoost: starting training.

[19107] signal (11.2): Segmentation fault: 11
in expression starting at REPL[53]:1
_ZN7xgboost4tree20CommonRowPartitioner14UpdatePositionIhLb0ELb0ENS0_14CPUExpandEntryEEEvPKNS_7ContextERKNS_16GHistIndexMatrixERKNS_6common12ColumnMatrixERKNSt3__16vectorIT2_NSE_9allocatorISG_EEEEPKNS_7RegTreeE at /Users/eph/.julia/artifacts/0079d93a46694d4e5e45f4e0b6bd6a35e24f4346/lib/libxgboost.dylib (unknown line)
_ZN7xgboost4tree20CommonRowPartitioner14UpdatePositionINS0_14CPUExpandEntryEEEvPKNS_7ContextERKNS_16GHistIndexMatrixERKNSt3__16vectorIT_NSA_9allocatorISC_EEEEPKNS_7RegTreeE at /Users/eph/.julia/artifacts/0079d93a46694d4e5e45f4e0b6bd6a35e24f4346/lib/libxgboost.dylib (unknown line)
_ZN7xgboost4tree11HistUpdater14UpdatePositionEPNS_7DMatrixEPKNS_7RegTreeERKNSt3__16vectorINS0_14CPUExpandEntryENS7_9allocatorIS9_EEEE at /Users/eph/.julia/artifacts/0079d93a46694d4e5e45f4e0b6bd6a35e24f4346/lib/libxgboost.dylib (unknown line)
_ZN7xgboost4tree10UpdateTreeINS0_14CPUExpandEntryENS0_11HistUpdaterEEEvPNS_6common7MonitorENS_6linalg10TensorViewIKNS_6detail20GradientPairInternalIfEELi2EEEPT0_PNS_7DMatrixEPKNS0_10TrainParamEPNS_16HostDeviceVectorIiEEPNS_7RegTreeE at /Users/eph/.julia/artifacts/0079d93a46694d4e5e45f4e0b6bd6a35e24f4346/lib/libxgboost.dylib (unknown line)
_ZN7xgboost4tree17QuantileHistMaker6UpdateEPKNS0_10TrainParamEPNS_16HostDeviceVectorINS_6detail20GradientPairInternalIfEEEEPNS_7DMatrixENS_6common4SpanINS5_IiEELm18446744073709551615EEERKNSt3__16vectorIPNS_7RegTreeENSH_9allocatorISK_EEEE at /Users/eph/.julia/artifacts/0079d93a46694d4e5e45f4e0b6bd6a35e24f4346/lib/libxgboost.dylib (unknown line)
_ZN7xgboost3gbm6GBTree13BoostNewTreesEPNS_16HostDeviceVectorINS_6detail20GradientPairInternalIfEEEEPNS_7DMatrixEiPNSt3__16vectorINS2_IiEENSA_9allocatorISC_EEEEPNSB_INSA_10unique_ptrINS_7RegTreeENSA_14default_deleteISI_EEEENSD_ISL_EEEE at /Users/eph/.julia/artifacts/0079d93a46694d4e5e45f4e0b6bd6a35e24f4346/lib/libxgboost.dylib (unknown line)
_ZN7xgboost3gbm6GBTree7DoBoostEPNS_7DMatrixEPNS_16HostDeviceVectorINS_6detail20GradientPairInternalIfEEEEPNS_20PredictionCacheEntryEPKNS_11ObjFunctionE at /Users/eph/.julia/artifacts/0079d93a46694d4e5e45f4e0b6bd6a35e24f4346/lib/libxgboost.dylib (unknown line)
_ZN7xgboost11LearnerImpl13UpdateOneIterEiNSt3__110shared_ptrINS_7DMatrixEEE at /Users/eph/.julia/artifacts/0079d93a46694d4e5e45f4e0b6bd6a35e24f4346/lib/libxgboost.dylib (unknown line)
XGBoosterUpdateOneIter at /Users/eph/.julia/artifacts/0079d93a46694d4e5e45f4e0b6bd6a35e24f4346/lib/libxgboost.dylib (unknown line)
XGBoosterUpdateOneIter at /Users/eph/.julia/packages/XGBoost/nqMqQ/src/Lib.jl:282 [inlined]
xgbcall at /Users/eph/.julia/packages/XGBoost/nqMqQ/src/Lib.jl:25 [inlined]
#updateone!#72 at /Users/eph/.julia/packages/XGBoost/nqMqQ/src/booster.jl:374
updateone! at /Users/eph/.julia/packages/XGBoost/nqMqQ/src/booster.jl:369 [inlined]
#update!#77 at /Users/eph/.julia/packages/XGBoost/nqMqQ/src/booster.jl:446
update! at /Users/eph/.julia/packages/XGBoost/nqMqQ/src/booster.jl:429 [inlined]
#xgboost#82 at /Users/eph/.julia/packages/XGBoost/nqMqQ/src/booster.jl:602
xgboost at /Users/eph/.julia/packages/XGBoost/nqMqQ/src/booster.jl:579
unknown function (ip: 0x2a35bc457)
_jl_invoke at /Users/julia/.julia/scratchspaces/a66863c6-20e8-4ff4-8a62-49f30b1f605e/agent-cache/default-honeycrisp-HL2F7YQ3XH.0/build/default-honeycrisp-HL2F7YQ3XH-0/julialang/julia-release-1-dot-10/src/gf.c:0 [inlined]
ijl_apply_generic at /Users/julia/.julia/scratchspaces/a66863c6-20e8-4ff4-8a62-49f30b1f605e/agent-cache/default-honeycrisp-HL2F7YQ3XH.0/build/default-honeycrisp-HL2F7YQ3XH-0/julialang/julia-release-1-dot-10/src/gf.c:3076
fit at /Users/eph/.julia/packages/MLJXGBoostInterface/uFARS/src/MLJXGBoostInterface.jl:168
unknown function (ip: 0x2a34bc71f)
_jl_invoke at /Users/julia/.julia/scratchspaces/a66863c6-20e8-4ff4-8a62-49f30b1f605e/agent-cache/default-honeycrisp-HL2F7YQ3XH.0/build/default-honeycrisp-HL2F7YQ3XH-0/julialang/julia-release-1-dot-10/src/gf.c:0 [inlined]
ijl_apply_generic at /Users/julia/.julia/scratchspaces/a66863c6-20e8-4ff4-8a62-49f30b1f605e/agent-cache/default-honeycrisp-HL2F7YQ3XH.0/build/default-honeycrisp-HL2F7YQ3XH-0/julialang/julia-release-1-dot-10/src/gf.c:3076
jl_apply at /Users/julia/.julia/scratchspaces/a66863c6-20e8-4ff4-8a62-49f30b1f605e/agent-cache/default-honeycrisp-HL2F7YQ3XH.0/build/default-honeycrisp-HL2F7YQ3XH-0/julialang/julia-release-1-dot-10/src/./julia.h:1982 [inlined]
do_apply at /Users/julia/.julia/scratchspaces/a66863c6-20e8-4ff4-8a62-49f30b1f605e/agent-cache/default-honeycrisp-HL2F7YQ3XH.0/build/default-honeycrisp-HL2F7YQ3XH-0/julialang/julia-release-1-dot-10/src/builtins.c:768
#fit_only!#57 at /Users/eph/.julia/packages/MLJBase/iIhiI/src/machines.jl:681
fit_only! at /Users/eph/.julia/packages/MLJBase/iIhiI/src/machines.jl:607 [inlined]
#fit_only!#62 at /Users/eph/.julia/packages/MLJBase/iIhiI/src/machines.jl:752
fit_only! at /Users/eph/.julia/packages/MLJBase/iIhiI/src/machines.jl:735 [inlined]
#80 at /Users/eph/.julia/packages/MLJBase/iIhiI/src/composition/learning_networks/nodes.jl:235
unknown function (ip: 0x2a2f4c0bb)
_jl_invoke at /Users/julia/.julia/scratchspaces/a66863c6-20e8-4ff4-8a62-49f30b1f605e/agent-cache/default-honeycrisp-HL2F7YQ3XH.0/build/default-honeycrisp-HL2F7YQ3XH-0/julialang/julia-release-1-dot-10/src/gf.c:0 [inlined]
ijl_apply_generic at /Users/julia/.julia/scratchspaces/a66863c6-20e8-4ff4-8a62-49f30b1f605e/agent-cache/default-honeycrisp-HL2F7YQ3XH.0/build/default-honeycrisp-HL2F7YQ3XH-0/julialang/julia-release-1-dot-10/src/gf.c:3076
jl_apply at /Users/julia/.julia/scratchspaces/a66863c6-20e8-4ff4-8a62-49f30b1f605e/agent-cache/default-honeycrisp-HL2F7YQ3XH.0/build/default-honeycrisp-HL2F7YQ3XH-0/julialang/julia-release-1-dot-10/src/./julia.h:1982 [inlined]
start_task at /Users/julia/.julia/scratchspaces/a66863c6-20e8-4ff4-8a62-49f30b1f605e/agent-cache/default-honeycrisp-HL2F7YQ3XH.0/build/default-honeycrisp-HL2F7YQ3XH-0/julialang/julia-release-1-dot-10/src/task.c:1238
Allocations: 41540566 (Pool: 41501312; Big: 39254); GC: 43
zsh: segmentation fault  julia --project

From that stacktrace, the last calls before we get into the C code are:

XGBoosterUpdateOneIter at /Users/eph/.julia/packages/XGBoost/nqMqQ/src/Lib.jl:282 [inlined]
xgbcall at /Users/eph/.julia/packages/XGBoost/nqMqQ/src/Lib.jl:25 [inlined]
#updateone!#72 at /Users/eph/.julia/packages/XGBoost/nqMqQ/src/booster.jl:374
updateone! at /Users/eph/.julia/packages/XGBoost/nqMqQ/src/booster.jl:369 [inlined]
#update!#77 at /Users/eph/.julia/packages/XGBoost/nqMqQ/src/booster.jl:446
update! at /Users/eph/.julia/packages/XGBoost/nqMqQ/src/booster.jl:429 [inlined]
#xgboost#82 at /Users/eph/.julia/packages/XGBoost/nqMqQ/src/booster.jl:602
xgboost at /Users/eph/.julia/packages/XGBoost/nqMqQ/src/booster.jl:579

which is why I filed it here rather than on MLJXGBoostInterface.jl.

I'm on v2.5.1 of XGBoost.jl, Julia v1.10.2,

julia> versioninfo()
Julia Version 1.10.2
Commit bd47eca2c8a (2024-03-01 10:14 UTC)
Build Info:
  Official https://julialang.org/ release
Platform Info:
  OS: macOS (arm64-apple-darwin22.4.0)
  CPU: 8 × Apple M1
  WORD_SIZE: 64
  LIBM: libopenlibm
  LLVM: libLLVM-15.0.7 (ORCJIT, apple-m1)
Threads: 4 default, 0 interactive, 2 GC (on 4 virtual cores)
Environment:
  JULIA_NUM_THREADS = 4
  JULIA_PKG_SERVER_REGISTRY_PREFERENCE = eager
@ericphanson
Copy link
Author

ericphanson commented Apr 6, 2024 •
edited
Loading

BTW in my actual code, I generally don't get a stacktrace, just

[18447] signal (11.2): Segmentation fault: 11
in expression starting at my_julia_code/fit.jl:42
Allocations: 243091803 (Pool: 242670917; Big: 420886); GC: 102
zsh: segmentation fault  julia --project

Edit: just got again, this time with maybe a bit more info, that there's a double-free:

julia(20662,0x325333000) malloc: Double free of object 0x2ad42abc0

[20662] signal (11.2): Segmentation fault: 11
in expression starting at more_julia_code/fit.jl:39
Allocations: 308241693 (Pool: 307688023; Big: 553670); GC: 114
julia(20662,0x325333000) malloc: *** set a breakpoint in malloc_error_break to debug
zsh: segmentation fault  julia --project

@ericphanson
Copy link
Author

I started julia under lldb and was able to repro, but I don't actually know what I'm doing, so not sure I found anything helpful.

Process 22988 stopped
* thread #22, stop reason = EXC_BAD_ACCESS (code=1, address=0x62e4bb3b0)
    frame #0: 0x00000002b2122d98 libxgboost.dylib`xgboost::common::QuantileSketchTemplate<float, float, xgboost::common::WQSummary<float, float> >::PushTemp() + 360
libxgboost.dylib`xgboost::common::QuantileSketchTemplate<float, float, xgboost::common::WQSummary<float, float> >::PushTemp:
->  0x2b2122d98 <+360>: stur   x1, [x16, #-0x20]
    0x2b2122d9c <+364>: add    x4, x1, x15
    0x2b2122da0 <+368>: stur   x2, [x16, #-0x10]
    0x2b2122da4 <+372>: add    x1, x1, x12
Target 0: (julia) stopped.
(lldb) thread backtrace
* thread #22, stop reason = EXC_BAD_ACCESS (code=1, address=0x62e4bb3b0)
  * frame #0: 0x00000002b2122d98 libxgboost.dylib`xgboost::common::QuantileSketchTemplate<float, float, xgboost::common::WQSummary<float, float> >::PushTemp() + 360
    frame #1: 0x00000002b2122bb4 libxgboost.dylib`xgboost::common::QuantileSketchTemplate<float, float, xgboost::common::WQSummary<float, float> >::Push(float, float) + 372
    frame #2: 0x00000002b2133ab8 libxgboost.dylib`.omp_outlined..118 + 360
    frame #3: 0x0000000172ee6b0c libomp.dylib`__kmp_invoke_microtask + 156
    frame #4: 0x0000000172e85a0c libomp.dylib`__kmp_invoke_task_func + 332
    frame #5: 0x0000000172e84914 libomp.dylib`__kmp_launch_thread + 420
    frame #6: 0x0000000172ec7d58 libomp.dylib`__kmp_launch_worker(void*) + 280
    frame #7: 0x00000001814a3fa8 libsystem_pthread.dylib`_pthread_start + 148
(lldb) thread list
Process 22988 stopped
  thread #1: tid = 0xd2916d, 0x00000002b2123010 libxgboost.dylib`void std::__1::__sort<std::__1::__less<xgboost::common::WQSummary<float, float>::Queue::QEntry, xgboost::common::WQSummary<float, float>::Queue::QEntry>&, xgboost::common::WQSummary<float, float>::Queue::QEntry*>(xgboost::common::WQSummary<float, float>::Queue::QEntry*, xgboost::common::WQSummary<float, float>::Queue::QEntry*, std::__1::__less<xgboost::common::WQSummary<float, float>::Queue::QEntry, xgboost::common::WQSummary<float, float>::Queue::QEntry>&) + 224, queue = 'com.apple.main-thread'
  thread #2: tid = 0xd29188, 0x000000018146a060 libsystem_kernel.dylib`kevent + 8
  thread #3: tid = 0xd29189, 0x0000000181463f14 libsystem_kernel.dylib`mach_msg2_trap + 8
  thread #4: tid = 0xd2918a, 0x0000000181467710 libsystem_kernel.dylib`__psynch_cvwait + 8
  thread #5: tid = 0xd2918b, 0x0000000181467710 libsystem_kernel.dylib`__psynch_cvwait + 8
  thread #6: tid = 0xd2918c, 0x0000000181467710 libsystem_kernel.dylib`__psynch_cvwait + 8
  thread #7: tid = 0xd2918d, 0x0000000181467710 libsystem_kernel.dylib`__psynch_cvwait + 8
  thread #8: tid = 0xd2918e, 0x0000000181467710 libsystem_kernel.dylib`__psynch_cvwait + 8
  thread #9: tid = 0xd2918f, 0x0000000181467710 libsystem_kernel.dylib`__psynch_cvwait + 8
  thread #10: tid = 0xd29190, 0x0000000181467710 libsystem_kernel.dylib`__psynch_cvwait + 8
  thread #13: tid = 0xd29232, 0x0000000181463f14 libsystem_kernel.dylib`mach_msg2_trap + 8
  thread #16: tid = 0xd2935f, 0x0000000181463f14 libsystem_kernel.dylib`mach_msg2_trap + 8
  thread #21: tid = 0xd2936f, 0x0000000000000000
* thread #22: tid = 0xd295b2, 0x00000002b2122d98 libxgboost.dylib`xgboost::common::QuantileSketchTemplate<float, float, xgboost::common::WQSummary<float, float> >::PushTemp() + 360, stop reason = EXC_BAD_ACCESS (code=1, address=0x62e4bb3b0)
  thread #23: tid = 0xd295b3, 0x00000002b212302c libxgboost.dylib`void std::__1::__sort<std::__1::__less<xgboost::common::WQSummary<float, float>::Queue::QEntry, xgboost::common::WQSummary<float, float>::Queue::QEntry>&, xgboost::common::WQSummary<float, float>::Queue::QEntry*>(xgboost::common::WQSummary<float, float>::Queue::QEntry*, xgboost::common::WQSummary<float, float>::Queue::QEntry*, std::__1::__less<xgboost::common::WQSummary<float, float>::Queue::QEntry, xgboost::common::WQSummary<float, float>::Queue::QEntry>&) + 252
  thread #24: tid = 0xd295b4, 0x00000001811a2484 dyld`dyld3::MachOLoaded::findClosestSymbol(unsigned long long, char const**, unsigned long long*) const + 552
(lldb) thread backtrace all
  thread #1, queue = 'com.apple.main-thread'
    frame #0: 0x00000002b2123010 libxgboost.dylib`void std::__1::__sort<std::__1::__less<xgboost::common::WQSummary<float, float>::Queue::QEntry, xgboost::common::WQSummary<float, float>::Queue::QEntry>&, xgboost::common::WQSummary<float, float>::Queue::QEntry*>(xgboost::common::WQSummary<float, float>::Queue::QEntry*, xgboost::common::WQSummary<float, float>::Queue::QEntry*, std::__1::__less<xgboost::common::WQSummary<float, float>::Queue::QEntry, xgboost::common::WQSummary<float, float>::Queue::QEntry>&) + 224
    frame #1: 0x00000002b2123248 libxgboost.dylib`void std::__1::__sort<std::__1::__less<xgboost::common::WQSummary<float, float>::Queue::QEntry, xgboost::common::WQSummary<float, float>::Queue::QEntry>&, xgboost::common::WQSummary<float, float>::Queue::QEntry*>(xgboost::common::WQSummary<float, float>::Queue::QEntry*, xgboost::common::WQSummary<float, float>::Queue::QEntry*, std::__1::__less<xgboost::common::WQSummary<float, float>::Queue::QEntry, xgboost::common::WQSummary<float, float>::Queue::QEntry>&) + 792
    frame #2: 0x00000002b2122b10 libxgboost.dylib`xgboost::common::QuantileSketchTemplate<float, float, xgboost::common::WQSummary<float, float> >::Push(float, float) + 208
    frame #3: 0x00000002b2133ab8 libxgboost.dylib`.omp_outlined..118 + 360
    frame #4: 0x0000000172ee6b0c libomp.dylib`__kmp_invoke_microtask + 156
    frame #5: 0x0000000172e85a0c libomp.dylib`__kmp_invoke_task_func + 332
    frame #6: 0x0000000172e81400 libomp.dylib`__kmp_fork_call + 7936
    frame #7: 0x0000000172e72fa4 libomp.dylib`__kmpc_fork_call + 180
    frame #8: 0x00000002b2133448 libxgboost.dylib`void xgboost::common::SketchContainerImpl<xgboost::common::WQuantileSketch<float, float> >::PushRowPageImpl<xgboost::data::SparsePageAdapterBatch, xgboost::common::SketchContainerImpl<xgboost::common::WQuantileSketch<float, float> >::PushRowPage(xgboost::SparsePage const&, xgboost::MetaInfo const&, xgboost::common::Span<float const, 18446744073709551615ul>)::'lambda'(auto)>(auto const&, unsigned long, xgboost::common::OptionalWeights, unsigned long, unsigned long, bool, xgboost::common::SketchContainerImpl<xgboost::common::WQuantileSketch<float, float> >::PushRowPage(xgboost::SparsePage const&, xgboost::MetaInfo const&, xgboost::common::Span<float const, 18446744073709551615ul>)::'lambda'(auto)) + 248
    frame #9: 0x00000002b2132b0c libxgboost.dylib`xgboost::common::SketchContainerImpl<xgboost::common::WQuantileSketch<float, float> >::PushRowPage(xgboost::SparsePage const&, xgboost::MetaInfo const&, xgboost::common::Span<float const, 18446744073709551615ul>) + 988
    frame #10: 0x00000002b20fdb24 libxgboost.dylib`xgboost::common::SketchOnDMatrix(xgboost::Context const*, xgboost::DMatrix*, int, bool, xgboost::common::Span<float const, 18446744073709551615ul>) + 3812
    frame #11: 0x00000002b2199708 libxgboost.dylib`xgboost::GHistIndexMatrix::GHistIndexMatrix(xgboost::Context const*, xgboost::DMatrix*, int, double, bool, xgboost::common::Span<float const, 18446744073709551615ul>) + 216
    frame #12: 0x00000002b21bcf30 libxgboost.dylib`xgboost::data::SimpleDMatrix::GetGradientIndex(xgboost::Context const*, xgboost::BatchParam const&) + 1168
    frame #13: 0x00000002b2350d20 libxgboost.dylib`xgboost::tree::HistUpdater::InitData(xgboost::DMatrix*, xgboost::RegTree const*) + 208
    frame #14: 0x00000002b232395c libxgboost.dylib`void xgboost::tree::UpdateTree<xgboost::tree::CPUExpandEntry, xgboost::tree::HistUpdater>(xgboost::common::Monitor*, xgboost::linalg::TensorView<xgboost::detail::GradientPairInternal<float> const, 2>, xgboost::tree::HistUpdater*, xgboost::DMatrix*, xgboost::tree::TrainParam const*, xgboost::HostDeviceVector<int>*, xgboost::RegTree*) + 156
    frame #15: 0x00000002b2322098 libxgboost.dylib`xgboost::tree::QuantileHistMaker::Update(xgboost::tree::TrainParam const*, xgboost::HostDeviceVector<xgboost::detail::GradientPairInternal<float> >*, xgboost::DMatrix*, xgboost::common::Span<xgboost::HostDeviceVector<int>, 18446744073709551615ul>, std::__1::vector<xgboost::RegTree*, std::__1::allocator<xgboost::RegTree*> > const&) + 1704
    frame #16: 0x00000002b21e83c4 libxgboost.dylib`xgboost::gbm::GBTree::BoostNewTrees(xgboost::HostDeviceVector<xgboost::detail::GradientPairInternal<float> >*, xgboost::DMatrix*, int, std::__1::vector<xgboost::HostDeviceVector<int>, std::__1::allocator<xgboost::HostDeviceVector<int> > >*, std::__1::vector<std::__1::unique_ptr<xgboost::RegTree, std::__1::default_delete<xgboost::RegTree> >, std::__1::allocator<std::__1::unique_ptr<xgboost::RegTree, std::__1::default_delete<xgboost::RegTree> > > >*) + 2356
    frame #17: 0x00000002b21e2548 libxgboost.dylib`xgboost::gbm::GBTree::DoBoost(xgboost::DMatrix*, xgboost::HostDeviceVector<xgboost::detail::GradientPairInternal<float> >*, xgboost::PredictionCacheEntry*, xgboost::ObjFunction const*) + 744
    frame #18: 0x00000002b21ff0d8 libxgboost.dylib`xgboost::LearnerImpl::UpdateOneIter(int, std::__1::shared_ptr<xgboost::DMatrix>) + 808
    frame #19: 0x00000002b20d6e98 libxgboost.dylib`XGBoosterUpdateOneIter + 152
    frame #20: 0x000000030a5e40b8
    frame #21: 0x000000030a45c504
    frame #22: 0x0000000101cc1ef4 libjulia-internal.1.10.2.dylib`do_apply [inlined] jl_apply(args=<unavailable>, nargs=<unavailable>) at julia.h:1982:12 [opt]
    frame #23: 0x0000000101cc1ee4 libjulia-internal.1.10.2.dylib`do_apply(args=0x00000003553ff9f8, nargs=<unavailable>, iterate=0x0000000127532040) at builtins.c:768:26 [opt]
    frame #24: 0x00000003215e4c34
    frame #25: 0x0000000101cd2efc libjulia-internal.1.10.2.dylib`start_task [inlined] jl_apply(args=<unavailable>, nargs=1) at julia.h:1982:12 [opt]
    frame #26: 0x0000000101cd2ef0 libjulia-internal.1.10.2.dylib`start_task at task.c:1238:19 [opt]
  thread #2
    frame #0: 0x000000018146a060 libsystem_kernel.dylib`kevent + 8
    frame #1: 0x0000000101d139e0 libjulia-internal.1.10.2.dylib`signal_listener(arg=<unavailable>) at signals-unix.c:747:27 [opt]
    frame #2: 0x00000001814a3fa8 libsystem_pthread.dylib`_pthread_start + 148
  thread #3
    frame #0: 0x0000000181463f14 libsystem_kernel.dylib`mach_msg2_trap + 8
    frame #1: 0x0000000181476240 libsystem_kernel.dylib`mach_msg2_internal + 80
    frame #2: 0x000000018146cb78 libsystem_kernel.dylib`mach_msg_overwrite + 604
    frame #3: 0x000000018146bf08 libsystem_kernel.dylib`mach_msg_server + 356
    frame #4: 0x0000000101d125fc libjulia-internal.1.10.2.dylib`mach_segv_listener(arg=<unavailable>) at signals-mach.c:104:15 [opt]
    frame #5: 0x00000001814a3fa8 libsystem_pthread.dylib`_pthread_start + 148
  thread #4
    frame #0: 0x0000000181467710 libsystem_kernel.dylib`__psynch_cvwait + 8
    frame #1: 0x00000001814a4574 libsystem_pthread.dylib`_pthread_cond_wait + 1232
    frame #2: 0x0000000101d82390 libjulia-internal.1.10.2.dylib`uv_cond_wait(cond=0x00000001008cb5b8, mutex=0x00000001008cb578) at thread.c:883:7
    frame #3: 0x0000000101cfb2d0 libjulia-internal.1.10.2.dylib`ijl_task_get_next(trypoptask=0x0000000126898c50, q=0x000000010d150030, checkempty=0x00000001268a0940) at partr.c:509:17 [opt]
    frame #4: 0x0000000123f14728 sys.dylib`julia_poptask_75480 at task.jl:985
    frame #5: 0x0000000124ccfc40 sys.dylib`julia_wait_74761 at task.jl:994
    frame #6: 0x00000002b65f04f0
    frame #7: 0x0000000101cd1dfc libjulia-internal.1.10.2.dylib`jl_finish_task [inlined] jl_apply(args=<unavailable>, nargs=2) at julia.h:1982:12 [opt]
    frame #8: 0x0000000101cd1df4 libjulia-internal.1.10.2.dylib`jl_finish_task(t=0x000000034d35dc30) at task.c:320:13 [opt]
    frame #9: 0x0000000101cd2f74 libjulia-internal.1.10.2.dylib`start_task at task.c:1249:5 [opt]
  thread #5
    frame #0: 0x0000000181467710 libsystem_kernel.dylib`__psynch_cvwait + 8
    frame #1: 0x00000001814a4574 libsystem_pthread.dylib`_pthread_cond_wait + 1232
    frame #2: 0x0000000101d82390 libjulia-internal.1.10.2.dylib`uv_cond_wait(cond=0x000000010400b3b8, mutex=0x000000010400b378) at thread.c:883:7
    frame #3: 0x0000000101cfb2d0 libjulia-internal.1.10.2.dylib`ijl_task_get_next(trypoptask=0x0000000126898c50, q=0x000000010d148030, checkempty=0x00000001268a0940) at partr.c:509:17 [opt]
    frame #4: 0x0000000123f14728 sys.dylib`julia_poptask_75480 at task.jl:985
    frame #5: 0x0000000124ccfc40 sys.dylib`julia_wait_74761 at task.jl:994
    frame #6: 0x00000002b65f04f0
    frame #7: 0x0000000101cd1dfc libjulia-internal.1.10.2.dylib`jl_finish_task [inlined] jl_apply(args=<unavailable>, nargs=2) at julia.h:1982:12 [opt]
    frame #8: 0x0000000101cd1df4 libjulia-internal.1.10.2.dylib`jl_finish_task(t=0x000000034d35ddc0) at task.c:320:13 [opt]
    frame #9: 0x0000000101cd2f74 libjulia-internal.1.10.2.dylib`start_task at task.c:1249:5 [opt]
  thread #6
    frame #0: 0x0000000181467710 libsystem_kernel.dylib`__psynch_cvwait + 8
    frame #1: 0x00000001814a4574 libsystem_pthread.dylib`_pthread_cond_wait + 1232
    frame #2: 0x0000000101d82390 libjulia-internal.1.10.2.dylib`uv_cond_wait(cond=0x000000010480b3b8, mutex=0x000000010480b378) at thread.c:883:7
    frame #3: 0x0000000101cfb2d0 libjulia-internal.1.10.2.dylib`ijl_task_get_next(trypoptask=0x0000000126898c50, q=0x000000010d140030, checkempty=0x00000001268a0940) at partr.c:509:17 [opt]
    frame #4: 0x0000000123f14728 sys.dylib`julia_poptask_75480 at task.jl:985
    frame #5: 0x0000000124ccfc40 sys.dylib`julia_wait_74761 at task.jl:994
    frame #6: 0x00000002b65f04f0
    frame #7: 0x0000000101cd1dfc libjulia-internal.1.10.2.dylib`jl_finish_task [inlined] jl_apply(args=<unavailable>, nargs=2) at julia.h:1982:12 [opt]
    frame #8: 0x0000000101cd1df4 libjulia-internal.1.10.2.dylib`jl_finish_task(t=0x000000034d35df50) at task.c:320:13 [opt]
    frame #9: 0x0000000101cd2f74 libjulia-internal.1.10.2.dylib`start_task at task.c:1249:5 [opt]
  thread #7
    frame #0: 0x0000000181467710 libsystem_kernel.dylib`__psynch_cvwait + 8
    frame #1: 0x00000001814a4574 libsystem_pthread.dylib`_pthread_cond_wait + 1232
    frame #2: 0x0000000101d82390 libjulia-internal.1.10.2.dylib`uv_cond_wait(cond=0x000000010202e1d0, mutex=0x000000010202e190) at thread.c:883:7
    frame #3: 0x0000000101cfab58 libjulia-internal.1.10.2.dylib`jl_gc_mark_threadfun(arg=<unavailable>) at partr.c:135:13 [opt]
    frame #4: 0x00000001814a3fa8 libsystem_pthread.dylib`_pthread_start + 148
  thread #8
    frame #0: 0x0000000181467710 libsystem_kernel.dylib`__psynch_cvwait + 8
    frame #1: 0x00000001814a4574 libsystem_pthread.dylib`_pthread_cond_wait + 1232
    frame #2: 0x0000000120173804 libopenblas64_.dylib`blas_thread_server + 388
    frame #3: 0x00000001814a3fa8 libsystem_pthread.dylib`_pthread_start + 148
  thread #9
    frame #0: 0x0000000181467710 libsystem_kernel.dylib`__psynch_cvwait + 8
    frame #1: 0x00000001814a4574 libsystem_pthread.dylib`_pthread_cond_wait + 1232
    frame #2: 0x0000000120173804 libopenblas64_.dylib`blas_thread_server + 388
    frame #3: 0x00000001814a3fa8 libsystem_pthread.dylib`_pthread_start + 148
  thread #10
    frame #0: 0x0000000181467710 libsystem_kernel.dylib`__psynch_cvwait + 8
    frame #1: 0x00000001814a4574 libsystem_pthread.dylib`_pthread_cond_wait + 1232
    frame #2: 0x0000000120173804 libopenblas64_.dylib`blas_thread_server + 388
    frame #3: 0x00000001814a3fa8 libsystem_pthread.dylib`_pthread_start + 148
  thread #13
    frame #0: 0x0000000181463f14 libsystem_kernel.dylib`mach_msg2_trap + 8
    frame #1: 0x0000000181476240 libsystem_kernel.dylib`mach_msg2_internal + 80
    frame #2: 0x000000018146cb78 libsystem_kernel.dylib`mach_msg_overwrite + 604
    frame #3: 0x0000000181464290 libsystem_kernel.dylib`mach_msg + 24
    frame #4: 0x00000001815827e4 CoreFoundation`__CFRunLoopServiceMachPort + 160
    frame #5: 0x00000001815810c4 CoreFoundation`__CFRunLoopRun + 1208
    frame #6: 0x00000001815804b8 CoreFoundation`CFRunLoopRunSpecific + 612
    frame #7: 0x000000018160557c CoreFoundation`CFRunLoopRun + 64
    frame #8: 0x0000000101d883cc libjulia-internal.1.10.2.dylib`uv__cf_loop_runner(arg=0x00000001022f04c0) at fsevents.c:734:3
    frame #9: 0x00000001814a3fa8 libsystem_pthread.dylib`_pthread_start + 148
  thread #16
    frame #0: 0x0000000181463f14 libsystem_kernel.dylib`mach_msg2_trap + 8
    frame #1: 0x0000000181476240 libsystem_kernel.dylib`mach_msg2_internal + 80
    frame #2: 0x000000018146cb78 libsystem_kernel.dylib`mach_msg_overwrite + 604
    frame #3: 0x0000000181464290 libsystem_kernel.dylib`mach_msg + 24
    frame #4: 0x00000001815827e4 CoreFoundation`__CFRunLoopServiceMachPort + 160
    frame #5: 0x00000001815810c4 CoreFoundation`__CFRunLoopRun + 1208
    frame #6: 0x00000001815804b8 CoreFoundation`CFRunLoopRunSpecific + 612
    frame #7: 0x000000018160557c CoreFoundation`CFRunLoopRun + 64
    frame #8: 0x0000000195da10d8 DebugSymbols`SpotlightQueryThread(void*) + 580
    frame #9: 0x00000001814a3fa8 libsystem_pthread.dylib`_pthread_start + 148
  thread #21
    frame #0: 0x0000000000000000
* thread #22, stop reason = EXC_BAD_ACCESS (code=1, address=0x62e4bb3b0)
  * frame #0: 0x00000002b2122d98 libxgboost.dylib`xgboost::common::QuantileSketchTemplate<float, float, xgboost::common::WQSummary<float, float> >::PushTemp() + 360
    frame #1: 0x00000002b2122bb4 libxgboost.dylib`xgboost::common::QuantileSketchTemplate<float, float, xgboost::common::WQSummary<float, float> >::Push(float, float) + 372
    frame #2: 0x00000002b2133ab8 libxgboost.dylib`.omp_outlined..118 + 360
    frame #3: 0x0000000172ee6b0c libomp.dylib`__kmp_invoke_microtask + 156
    frame #4: 0x0000000172e85a0c libomp.dylib`__kmp_invoke_task_func + 332
    frame #5: 0x0000000172e84914 libomp.dylib`__kmp_launch_thread + 420
    frame #6: 0x0000000172ec7d58 libomp.dylib`__kmp_launch_worker(void*) + 280
    frame #7: 0x00000001814a3fa8 libsystem_pthread.dylib`_pthread_start + 148
  thread #23
    frame #0: 0x00000002b212302c libxgboost.dylib`void std::__1::__sort<std::__1::__less<xgboost::common::WQSummary<float, float>::Queue::QEntry, xgboost::common::WQSummary<float, float>::Queue::QEntry>&, xgboost::common::WQSummary<float, float>::Queue::QEntry*>(xgboost::common::WQSummary<float, float>::Queue::QEntry*, xgboost::common::WQSummary<float, float>::Queue::QEntry*, std::__1::__less<xgboost::common::WQSummary<float, float>::Queue::QEntry, xgboost::common::WQSummary<float, float>::Queue::QEntry>&) + 252
    frame #1: 0x00000002b2123248 libxgboost.dylib`void std::__1::__sort<std::__1::__less<xgboost::common::WQSummary<float, float>::Queue::QEntry, xgboost::common::WQSummary<float, float>::Queue::QEntry>&, xgboost::common::WQSummary<float, float>::Queue::QEntry*>(xgboost::common::WQSummary<float, float>::Queue::QEntry*, xgboost::common::WQSummary<float, float>::Queue::QEntry*, std::__1::__less<xgboost::common::WQSummary<float, float>::Queue::QEntry, xgboost::common::WQSummary<float, float>::Queue::QEntry>&) + 792
    frame #2: 0x00000002b2122b10 libxgboost.dylib`xgboost::common::QuantileSketchTemplate<float, float, xgboost::common::WQSummary<float, float> >::Push(float, float) + 208
    frame #3: 0x00000002b2133ab8 libxgboost.dylib`.omp_outlined..118 + 360
    frame #4: 0x0000000172ee6b0c libomp.dylib`__kmp_invoke_microtask + 156
    frame #5: 0x0000000172e85a0c libomp.dylib`__kmp_invoke_task_func + 332
    frame #6: 0x0000000172e84914 libomp.dylib`__kmp_launch_thread + 420
    frame #7: 0x0000000172ec7d58 libomp.dylib`__kmp_launch_worker(void*) + 280
    frame #8: 0x00000001814a3fa8 libsystem_pthread.dylib`_pthread_start + 148
  thread #24
    frame #0: 0x00000001811a2484 dyld`dyld3::MachOLoaded::findClosestSymbol(unsigned long long, char const**, unsigned long long*) const + 552
    frame #1: 0x000000018118215c dyld`dyld4::APIs::dladdr(void const*, dl_info*) + 212
    frame #2: 0x000000018134d974 libsystem_c.dylib`backtrace_symbols + 120
    frame #3: 0x00000002b20b6a24 libxgboost.dylib`dmlc::StackTrace(unsigned long, unsigned long) + 372
    frame #4: 0x00000002b20b680c libxgboost.dylib`dmlc::LogMessageFatal::~LogMessageFatal() + 124
    frame #5: 0x00000002b2124170 libxgboost.dylib`xgboost::common::WQSummary<float, float>::SetCombine(xgboost::common::WQSummary<float, float> const&, xgboost::common::WQSummary<float, float> const&) + 1296
    frame #6: 0x00000002b2122ea8 libxgboost.dylib`xgboost::common::QuantileSketchTemplate<float, float, xgboost::common::WQSummary<float, float> >::PushTemp() + 632
    frame #7: 0x00000002b2122bb4 libxgboost.dylib`xgboost::common::QuantileSketchTemplate<float, float, xgboost::common::WQSummary<float, float> >::Push(float, float) + 372
    frame #8: 0x00000002b2133ab8 libxgboost.dylib`.omp_outlined..118 + 360
    frame #9: 0x0000000172ee6b0c libomp.dylib`__kmp_invoke_microtask + 156
    frame #10: 0x0000000172e85a0c libomp.dylib`__kmp_invoke_task_func + 332
    frame #11: 0x0000000172e84914 libomp.dylib`__kmp_launch_thread + 420
    frame #12: 0x0000000172ec7d58 libomp.dylib`__kmp_launch_worker(void*) + 280
    frame #13: 0x00000001814a3fa8 libsystem_pthread.dylib`_pthread_start + 148

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@ericphanson