Compiling GENIE 3.6.2 with INCL++ v6.33.1

[ghkareem]

I am having some trouble compiling the latest release of GENIE with the latest INCL++ release and wondered if this is just me or if there is a compatibility issue for INCL++ > 5?

Thank you for any help!

[LiangLiu212]

Hi, @ghkareem ,

Currently, we use a variant of INCL++ package for GENIE and it is different from the INCL++ standalone. So, you can not compile GENIE with the standalone INCL++ package now.

I'm developing a new interface for GENIE to use the native INCL++ code. But it will take a while (~half year).

Please let me know if you have more questions.

Best regards!

Liang Liu

[ghkareem]

Hi @LiangLiu212 thanks so much for this information!

Is this variant possible to download/request from somewhere? Thanks for any information :)

Many thanks!
Kareem

[ghkareem]

Sorry Liang Liu I should have added but basically I want to run gevgen with the G18_10c_02_11b tune, but I think in order to do this I have to configure with INCL++ somehow - I also found that in release 3.2 there was not a precompiled cross section so I assume that means using gmkspl to create cross sections. Is this also correct?

Once again I am grateful for your help - best regards!
Kareem

[nusense]

@ghkareem

I softly question your continued use of GENIE v3_02_XX which is two releases behind (circa November 2022 at the least). Is there a reason not to use v3_06_00 or v3_06_02? One would need to study the release notes to see if there are significant differences in your use case.

Unfortunately the GENIE collaboration is not at liberty to distribute the modified INCL++ code that is necessary to run it in the existing releases of GENIE due to licensing issues. I don't know what experiment you're working on, so it's hard to give general guidance. Do you have access to Fermilab resources where we do have a version built? Note, the builds of INCL++ prior to the one used in v3_06_00 include a bug in our modifications to INCL++ that should be avoided. The correctly modified version of INCL++ might be included in some versions of the Fermilab v3_04_02 builds as well.

Also at this time we don't have pre-computed splines for the G18_10X_02_11b tunes for v3_06_XX, though we do have them for v3_04_XX and v3_02_XX (though see above note about the INCL++ builds) . What is available can be found at:
v3_06_XX
v3_04_XX
v3_02_XX

Despite the fact that the FSI model choice doesn't affect the cross-sections, the spline file may need a trivial adjustment for different FSI variants due to how the tune information is encoded into the tune name and the internal matching that GENIE performs. I can describe the structure of these tar files if you're not using the UPS packaging system. I can also describe how to make the FSI adjustment if you're interested.

If there's sufficient interest I can produce v3_06_XX splines for tunes beyond what's currently available. Generating splines for a limited number of isotopes and neutrino flavors using gmkspl is a doable project. For more expansive sets, it generally takes significantly more resources than can easily be mustered by users. The precomputed splines cover all six neutrino flavors on 125 isotopes (51 in the file that is normally used) for the energy range [0,1000] GeV using 250 logarithmically spaced knots. These are generated using grid resources. I don't know whether this configuration covers your use case.

I hope this helps clarify the situation.

[ghkareem]

Hi Robert, and thanks for your message,

My comment about 3.2 was to say it appears the first instance I could find mention of the G18_10c_02_11b tune; I'm actually trying to set up 3.6.2 with INCL. Thanks to your suggestions I could find the variant on the UPS system, and I managed to compile 3.6.2 with it :)

Actually, It would be super helpful to know how to perform the FSI adjustment! I'm only interested in neutrinos with water target for testing with IceCube; I have access to submit jobs on my local cluster so if you could inform me how I could generate the splines (I see in the manual some discussion of submitting jobs? also something along the lines of gmkspl -p 12,14,16,-12,-14,-16 -t 1000010010, 1000080160 -n 250 -e 1000 --seed 12345 --tune G18_10c_02_11b should work?)

Thank you for your help in this matter!

[nusense]

@ghkareem

Okay, I'm glad you're not stuck in the past. Too many groups freeze out and never update and then it becomes difficult to support them.

In principle what you have there for using gmkspl looks like it should work. Except I think there's a space between your two elements, and I'd suggest using the -o flag to name the file something that reflects its contents. I think the --seed is unnecessary (there's a default), but it doesn't hurt. Hidden in there is an implicit calculation of cross-sections on free protons and neutrons (though that makes the Hydrogen calculation trivial). That and the fact that you're doing all 6 flavors might make the job take too long for your resources. In which case, I might suggest doing each flavor separately and then using gspladd to combine them.

The trick to making a cross-section spline file compatible with a FSI that it wasn't generated for is to modify the name that's embedded at the front of the XML (i.e. text) file. The 7th line should look something like:

 <genie_tune name="G18_10c_02_11b">

And in this case you would be changing the "c" to something else and saving the file to a different name (so you can easily switch between tunes without further edits). The sed command will let you do this easily without using an interactive editor if you're comfortable with that.

If you'd like I could fire off the grid jobs to generate G18_10a_02_11b for general use by all. It would take a couple of days (depending on whether there a failed intermediate jobs that need to be resubmitted) to generate our usual set. Then you could modify those results, as described above, changing "a" to "c". Actually why don't I just do that. If loading the larger set of isotopes than you need seems wasteful (it is), then you can also use the awk script reduce_gxspl.awk contained in the tar file as a template on how to further reduce the isotopes from gxspl-NUsmall.xml into a new file, e.g. gxspl-H2O-G18_10a_02_11b.xml and gxspl-H2O-G18_10c_02_11b.xml

[ghkareem]

I actually submitted some jobs yesterday and I managed to get some splines for nue(bar)+numu(bar) but nutau seemed to fail, so am rerunning; thanks though! :)

Of course if it helps other users that that is great, and thanks for telling me the trick too Robert, am grateful for the help! :)

[nusense]

@ghkareem
In case your jobs didn't complete, or you've decided that you need more isotopes, my efforts to produce splines for G18_10a_02_11b (note the a, not your desired c) have completed and I've uploaded the tarball to: https://scisoft.fnal.gov/scisoft/packages/genie_xsec/v3_06_00/genie_xsec-3.06.00-noarch-G1810a0211b-k250-e1000.tar.bz2

Best of luck. Unless there's further need for discussion I'll probably close this issue in the next few days.

[ghkareem]

@nusense Thanks so much! Actually I was going to ask, but my jobs for G18_10c_02_11b nue/nubar with hydrogen/oxygen finished and looks good, however numu/nutau seem to be stuck for G18_10c_02_11b; do we expect them to take longer due to the difference in flavour?

If helpful, I can share them too if it would be useful to others once completed
Many thanks and best wishes

[nusense]

@ghkareem Yes, sometimes (for reasons I don't quite understand) different flavors will take significantly different amounts of time for the same isotopes. For my case this time numubar on He3 took way longer than any of the other nu flavors on light isotopes combinations.

I think this has to do with the fact that some of the cross-section calculation is done by exploring the phase space with Monte Carlo methods to do some integrals and depending on conditions can take different amounts of time to converge. I've never tried to see if these long tail outliers are affected by perhaps a change in the random number seed.

[ghkareem]

@nusense thanks for the information, okay I'll hold in there for a while longer.

I might reopen the issue in a few days if something happens but otherwise I think its okay for now - just to say I really appreciate yours and @LiangLiu212 's help! :)

[ghkareem]

@nusense so it looks like overnight my jobs died with ERROR - Floating point invalid operation. particularly for numu+nutau on Oxygen.
I'm gonna have to resubmit anyway, so is there any way to run gmkspl on a GPU in the hopes that it speed up the calculation?
Any other suggestions would also be great - thanks again! :)

PS also (if its okay to add this as a separate issue), does gevgen have a way to change the principle neutrino direction from the z axis?

[nusense]

@ghkareem Sorry, GENIE doesn't at this point have any plans for running on GPUs (too complex, too many variants for our present manpower to support).

I'm a bit surprised that you're getting an FPE. I'm not sure I've ever seen that during my spline generation efforts. It seems odd.

The gevgen application does support adjusting the flux driver (histogram method only) parameters.

 -F
              Specifies direction,size,start point of histogram flux
              dircosx,dircosy,dircosz,Radius,spotx,spoty,spotz

where you can give 3,4, or 7 values (though it doesn't check properly so be sure only to give a consistent set). If you give the first three you override the default direction (0,0,1). If you give the next, then you override the default transverse size of the pencil beam -1 (which it takes as equivalent to 0). If you specify all 7 values, then you also override the position where the "target" is, with the default being (0,0,0).

I think this -F flag in gevgen is only applicable to histogram fluxes, and not to mono-energetic flux, or the power law flux. I believe the "functional form" and "vector file" counts as a "histogram" as long as the range is non-zero in width. So -e 10 is mono energetic, but -e 10,20 or even -e 10,10.0001 isn't. (sigh) The drivers are a bit inconsistent in their configuration which is why the -F flag isn't applied to the non-histogram drivers -- it would be nice to clean this up a bit, but there's little demand for it.

[ghkareem]

Hi Robert and thanks so much for this information! OK that is very helpful indeed, and I really appreciate this!
So in the end I could get jobs working for Hydrogen target successfully, but it looks like I am encountering this FPE quite frequently with jobs, so I just wanted to put this here if it provides some clue about the error.

Looking at the size of a completed job (~30M) it looks like the jobs die about 20% of the way through with an FPE, meaning I couldnt yet generate full splines for G18_10c_02_11b, but I've submitted again to try today so maybe it eventually makes it to the end(?)

Thanks again for any help/guidance with this error :)
Many thanks and best wishes,
Kareem

job_chunk_8442039.0_err.txt

job_chunk_8442039.0_out.txt

[nusense]

Just for reference, this failed at:

1755309505 NOTICE XSecSplLst : [s]  : Creating cross section spline using the algorithm:
Algorithm Key: genie::BergerSehgalCOHPiPXSec2015/Default - Owns Substruc: [false]
[-] Registry name: [genie::BergerSehgalCOHPiPXSec2015/Default_summary] - Write Status: [unlocked] - Inhibited Item Locking: [on] - # entries:  16
 |-----------------------------------------> CKM-Vcd [double] [unlocked] [l] : 0.22
 |-----------------------------------------> CKM-Vcs [double] [unlocked] [l] : 0.995
 |-----------------------------------------> CKM-Vud [double] [unlocked] [l] : 0.97417
 |-----------------------------------------> CKM-Vus [double] [unlocked] [l] : 0.2248
 |------------------------------------------> COH-Ma [double] [unlocked] [l] : 1
 |--------------------------------------> COH-Q2-max [double] [unlocked] [l] : 1
 |--------------------------------------> COH-Q2-min [double] [unlocked] [l] : 0
 |------------------------------------------> COH-Ro [double] [unlocked] [l] : 1
 |-------------------------------> COH-UseRSPionXSec   [bool] [unlocked] [l] : 0
 |---------------------------------------> COH-t-max [double] [unlocked] [l] : 0.25
 |------------------------------------> CabibboAngle [double] [unlocked] [l] : 0.22778
 |-------------------------------------> CommonParam [string] [unlocked] [l] : Coherent,CKM
 |--------------------------------> DIS-CC-XSecScale [double] [unlocked] [l] : 1.06221
 |--------------------------------> RES-CC-XSecScale [double] [unlocked] [l] : 0.83873
 |------------------------------------------> RES-Ma [double] [unlocked] [l] : 1.08896
 |---------------------------------> XSec-Integrator    [alg] [unlocked] [l] : genie::COHXSec/Default

1755309505 NOTICE XSecSplLst : [s]  : Energy threshold for current interaction = 0.247247 GeV
1755309505 NOTICE XSecSplLst : [s]  : xsec(E = 0.01) =  0 x 1E-38 cm^2, evaluated in 6.3686e-05 s
1755309505 NOTICE XSecSplLst : [s]  : xsec(E = 0.0574494) =  0 x 1E-38 cm^2, evaluated in 1.885e-06 s
1755309505 NOTICE XSecSplLst : [s]  : xsec(E = 0.104899) =  0 x 1E-38 cm^2, evaluated in 1.534e-06 s
1755309505 NOTICE XSecSplLst : [s]  : xsec(E = 0.152348) =  0 x 1E-38 cm^2, evaluated in 1.447e-06 s
1755309505 NOTICE XSecSplLst : [s]  : xsec(E = 0.199798) =  0 x 1E-38 cm^2, evaluated in 1.504e-06 s
1755309506 NOTICE XSecSplLst : [s]  : xsec(E = 0.247247) =  2.06085e-199 x 1E-38 cm^2, evaluated in 1.71929 s
1755309514 NOTICE XSecSplLst : [s]  : xsec(E = 0.255808) =  6.21817e-20 x 1E-38 cm^2, evaluated in 7.48509 s
1755309521 NOTICE XSecSplLst : [s]  : xsec(E = 0.264665) =  6.82511e-12 x 1E-38 cm^2, evaluated in 7.40476 s
1755309529 NOTICE XSecSplLst : [s]  : xsec(E = 0.273828) =  5.18652e-09 x 1E-38 cm^2, evaluated in 7.42433 s
1755309536 NOTICE XSecSplLst : [s]  : xsec(E = 0.283309) =  1.7093e-07 x 1E-38 cm^2, evaluated in 7.63963 s
1755309544 NOTICE XSecSplLst : [s]  : xsec(E = 0.293118) =  1.55718e-06 x 1E-38 cm^2, evaluated in 7.52388 s
1755309551 NOTICE XSecSplLst : [s] 
But I don't see anything special.