Problems with beta version of CLASS 3.0.0

[pstoecker]

As discussed during the CosmoBit meeting, there are some issues with the upcoming, yet unpublished, version of CLASS. In particular I have seen two issues so far in the context of the generic decayingDM models.

1. For combinations of lifetime and fraction that lead to very strong energy injection and are well excluded, CLASS happens to get stuck within the thermodynamics module. Unfortunately this is super sensitive to the specific input values such that I was not able so far to get a reproducible input values (i.e. the input values printed to cout in debug mode are not precise enough).
2. For very short lifetimes (1e11s - 1e13s), CLASS crashes within the thermodynamics module during a call of the ndf15 solver. As for the first issue, also this is basically non-reproducible as it also super sensitive to the precise inputs.

For the latter issue the unwinded stack reads:

[HP-envy:31315] *** Process received signal ***
[HP-envy:31315] Signal: Segmentation fault (11)
[HP-envy:31315] Signal code: Address not mapped (1)
[HP-envy:31315] Failing at address: 0x8
[HP-envy:31315] [ 0] /lib/x86_64-linux-gnu/libpthread.so.0(+0x12980)[0x7fbaca27e980]
[HP-envy:31315] [ 1] /work/gambit_master/Backends/installed/classy/3.0.0_beta/lib/classy.so(numjac+0x105b)[0x7fbac3b65fb7]
[HP-envy:31315] [ 2] /work/gambit_master/Backends/installed/classy/3.0.0_beta/lib/classy.so(evolver_ndf15+0xb1c)[0x7fbac3b5edaa]
[HP-envy:31315] [ 3] /work/gambit_master/Backends/installed/classy/3.0.0_beta/lib/classy.so(thermodynamics_reionization_evolve_with_tau+0x949)[0x7fbac3aa665d]
[HP-envy:31315] [ 4] /work/gambit_master/Backends/installed/classy/3.0.0_beta/lib/classy.so(thermodynamics_solve+0x60a)[0x7fbac3aa3710]
[HP-envy:31315] [ 5] /work/gambit_master/Backends/installed/classy/3.0.0_beta/lib/classy.so(thermodynamics_init+0x845)[0x7fbac3a9dc8a]
[HP-envy:31315] [ 6] /work/gambit_master/Backends/installed/classy/3.0.0_beta/lib/classy.so(+0x1e719)[0x7fbac39b9719]
[HP-envy:31315] [ 7] /work/gambit_master/Backends/installed/classy/3.0.0_beta/lib/classy.so(+0x1bf8d)[0x7fbac39b6f8d]
[HP-envy:31315] [ 8] /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0(PyObject_Call+0x43)[0x7fbac9e63913]
[HP-envy:31315] [ 9] /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0(PyEval_CallObjectWithKeywords+0x47)[0x7fbac9ecde37]
[HP-envy:31315] [10] ./gambit(+0xf934c8)[0x56261c4274c8]
[HP-envy:31315] [11] ./gambit(+0xebe6ca)[0x56261c3526ca]
[HP-envy:31315] [12] ./gambit(+0x13eae74)[0x56261c87ee74]
[HP-envy:31315] [13] ./gambit(+0x1453b87)[0x56261c8e7b87]
[HP-envy:31315] [14] ./gambit(+0x1a324b1)[0x56261cec64b1]
[HP-envy:31315] [15] ./gambit(+0x1aa95c1)[0x56261cf3d5c1]
[HP-envy:31315] [16] /work/gambit_master/ScannerBit/lib/libscanner_multinest_3.11.so(+0x3a7d5)[0x7fba8c0ac7d5]
[HP-envy:31315] [17] /work/gambit_master/ScannerBit/lib/libscanner_multinest_3.11.so(+0x37f32)[0x7fba8c0a9f32]
[HP-envy:31315] [18] /work/gambit_master/ScannerBit/lib/libscanner_multinest_3.11.so(_ZN6Gambit9MultiNest14LogLikeWrapper7LogLikeEPdii+0x80)[0x7fba8c0a063a]
[HP-envy:31315] [19] /work/gambit_master/ScannerBit/lib/libscanner_multinest_3.11.so(_ZN6Gambit9MultiNest16callback_loglikeEPdiiPv+0x32)[0x7fba8c0a04ef]
[HP-envy:31315] [20] /work/gambit_master/ScannerBit/installed/multinest/3.11/libnest3.so(__cnested_MOD_loglike_f+0x69)[0x7fba85a4b795]
[HP-envy:31315] [21] /work/gambit_master/ScannerBit/installed/multinest/3.11/libnest3.so(__nested_MOD_gen_initial_live+0x20ea)[0x7fba85a4718d]
[HP-envy:31315] [22] /work/gambit_master/ScannerBit/installed/multinest/3.11/libnest3.so(__nested_MOD_nestsample+0x1025)[0x7fba85a49449]
[HP-envy:31315] [23] /work/gambit_master/ScannerBit/installed/multinest/3.11/libnest3.so(__nested_MOD_nestrun+0x19a4)[0x7fba85a4b34c]
[HP-envy:31315] [24] /work/gambit_master/ScannerBit/installed/multinest/3.11/libnest3.so(run+0x251)[0x7fba85a4b616]
[HP-envy:31315] [25] /work/gambit_master/ScannerBit/lib/libscanner_multinest_3.11.so(_ZN60__gambit_plugin_multinest__t__scanner__v__3_11___namespace__11PLUGIN_MAINEv+0x113d)[0x7fba8c09fe1e]
[HP-envy:31315] [26] ./gambit(+0x178e75b)[0x56261cc2275b]
[HP-envy:31315] [27] ./gambit(+0x178acbe)[0x56261cc1ecbe]
[HP-envy:31315] [28] ./gambit(+0x1785e0e)[0x56261cc19e0e]
[HP-envy:31315] [29] ./gambit(+0x8a1c90)[0x56261bd35c90]
[HP-envy:31315] *** End of error message ***

such that I already started to search for the issue in the numjac function of the ndf15 solver of CLASS. In fact, the actual issue is that the function that is called by this evolver (through the function pointer *derivs) is returning nan such that an indexing variable is left uninitialised with a random value. Thus a forbidden address is accessed and the crash occurs. However, I have not further investigated why exactly the function behind the *derivs pointer (which is thermodynamics_derivs in this case) is producing a NaN.

To reproduce the issue, everything is on the CosmoBit_development branch and the isuue will occur when you execute either yaml_files_EnergyInjection/Decay_photon_zreio.yaml or `yaml_files_EnergyInjection/Decay_photon.yaml'. You should hit a crashing parameter point relatively soon.

For this to work, you need to compile at least CosmoBit as module. Furthermore, you need multinest and the following backends: make alterbbn darkages classy_3.0.0_beta plc plc_data.

@fruzsinaagocs, I tag you, as you were interested in helping out with this. In case you have questions, let me know.
I also tag @anderkve , as you wanted to be kept in the loop.

[pstoecker]

Note that the parameter ranges of lifetime and fraction within yaml_files_EnergyInjection/Decay_photon_zreio.yaml and yaml_files_EnergyInjection/Decay_photon.yamlare chosen such that the second issue occurs relatively fast. If you want to search for the reason for the first issue, you will need to use the full ranges that are currently commented in the files.

[fruzsinaagocs]

The 2. problem described above (seg faults for short lifetimes) is present in the recent CLASS 3.0.0 release, which uses HyRec instead of recfast, as well. Should I open a new issue for that, then maybe reference it here and close this one? What would you recommend?

[pstoecker]

I would keep the discussion here.

Do you know whether the segfault is still caused from within the numjac method of the ndf15 solver that class is using? Then it probably caused by one of the functions that make up the ode system to produce a NaN.

The quickest, but probably most efficient solution would be to check for this and let CLASS exit with an error. This error is then interpreted as invalid point on the GAMBIT side.

[fruzsinaagocs]

Yes, it's still numjac within ndf15:

==== backtrace (tid: 691107) ====
 0 0x0000000000055969 ucs_debug_print_backtrace()  ???:0
 1 0x0000000000012b20 .annobin_sigaction.c()  sigaction.c:0
 2 0x00000000001666c6 numjac()  ???:0
 3 0x0000000000160140 evolver_ndf15()  ???:0
 4 0x00000000000ded10 thermodynamics_solve()  ???:0
 5 0x00000000000e60f3 thermodynamics_init()  ???:0
 6 0x0000000000020fe2 __pyx_pf_6classy_5Class_20compute()  /home/552/fa7965/gambit/Backends/installed/classy/3.0.0/python/../python/classy.c:7271
 7 0x00000000000206ad __pyx_pw_6classy_5Class_21compute()  /home/552/fa7965/gambit/Backends/installed/classy/3.0.0/python/../python/classy.c:6676
 8 0x00000000000f1517 cfunction_call_varargs()  /home/900/z30_apps/builds/_UYaaG8i/1/nci/gadi-apps/python3/3.7.4/source/Python-3.7.4/build/../Objects/call.c:757
 9 0x00000000000f1517 PyCFunction_Call()  /home/900/z30_apps/builds/_UYaaG8i/1/nci/gadi-apps/python3/3.7.4/source/Python-3.7.4/build/../Objects/call.c:788
10 0x0000000001041ced pybind11::detail::simple_collector<(pybind11::return_value_policy)1>::call()  :0
11 0x0000000001c1b138 pybind11::detail::object_api<pybind11::detail::accessor<pybind11::detail::accessor_policies::str_attr> >::operator()<(pybind11::return_value_policy)1>()  :0
12 0x00000000021f3a01 Gambit::BackendIniBit::classy_3_0_0_init()  :0
13 0x000000000225aded Gambit::module_functor<void>::calculate()  :0
14 0x0000000002fa3dfa Gambit::DRes::DependencyResolver::calcObsLike()  :0
15 0x00000000030156dc Gambit::Likelihood_Container::main()  :0
16 0x000000000000cc15 Gambit::MultiNest::LogLikeWrapper::LogLike()  ???:0
17 0x000000000011689b cnested_mp_loglike_f_()  ???:0
18 0x0000000000113c7a nested_mp_gen_initial_live_()  ???:0
19 0x00000000000abf05 nested_mp_nestsample_()  ???:0
20 0x00000000000aaa5c nested_mp_nestrun_()  ???:0
21 0x0000000000116b29 run()  ???:0
22 0x000000000000e040 __gambit_plugin_multinest__t__scanner__v__3_11___namespace__::PLUGIN_MAIN()  ???:0
23 0x0000000002c0ac36 Gambit::Scanner::Scan_Manager::Run()  :0
24 0x000000000056e419 main()  :0
25 0x00000000000237b3 __libc_start_main()  ???:0
26 0x000000000056d4fe _start()  ???:0

[fruzsinaagocs]

I've put in a class_test(...) to check for the derivatives of thermodynamical variables being NaNs. However, now I had two different types of error pop up (so far):

1. A seg fault in numjac within ndf15, but this time in the perturbations module (I think I can get rid of this by moving the class_test(...) into numjac instead of the specific <module>_derivs function),
2. a "stepsize too small" error produced by ndf15 in the thermodynamics module - is this the same as your issue 1. above?

Btw, where should I put my changes to CLASS? Should I submit a PR here, or to the CLASS repo itself?

[pstoecker]

Ok the (new) first issue is very interesting. I think that moving the test into ndf15 is the best thing, also as a general solution. When we are happy with the result, we should contact Julien and let him know about the issue and should open a PR on the CLASS repo for this such that this is fixed in future releases of CLASS. Just for completion a similar check should be implemented to the rather inefficient rk solver, just in case a user ever decides to use this solver.

However, for the current version (3.0.0), it would be good if this can be added to the patch that we already apply. Could you do that? If not, just send me the diff for your particular fix and I will add this to the patch file. Since CLASS 3.0.0 only lives on CosmoBit_development there is no need for a PR, unless you prefer it that way. I am happy if you just commit it and let me know.

To the second issue: This happens quite often and it is totally normal. To my understanding, this issue refers to points with a strong energy injection such that the derivatives in the thermodynamics module are very steep such that the stepsize must be very tiny to resolve the derivative. One could try to fiddle a little bit with the precision parameter that controls the minimal stepsize, but you do not need to do this if you don't want to as these points are anyway excluded.

[fruzsinaagocs]

I've moved the NaN check into numjac, and pushed the new patch file to CosmoBit_development in c2811cd. With the patch, I only get the "stepsize too small" error, and - at least when using ndf15 as the evolver - no seg faults.

Let me know if you're happy with this, and we can potentially let Julien know.

[pstoecker]

Thanks a lot for taking care of this.

You have accidentally also included the fixes to the setup.py script of the python wrapper that needed to be applied on gadi on some other clusters that use the intel compilers and a modification to the Makefile that we actually apply by hand in the cmake step. The latter one has no negative effect whereas the former one breaks compilation with gcc. I have fixed both parts of your patch in 5457119.

I will have a look on this but so far it sounds to be a good solution such that we can let Julien know about this.

@fruzsinaagocs I assume that there is no further need to have the version 3.0.0_beta of classy. I deleted this version in commit 6cf427d. In case that you need this version back, you need to revert it (locally).

[pstoecker]

Seems like @lukashergt has discovered the ndf15 issue in a different context (see lesgourg/class_public#419)

I would suggest that we forward our fix to Julien as soon as possible.

[fruzsinaagocs]

PR to CLASS submitted: lesgourg/class_public#423

[pstoecker]

Thanks a lot.