TwoStream solver crashes with mismatch at restart

[maritsandstad]

Brief summary of bug

When restarting a specific case, the FatesRestartInterfaceMod sets_restart_vectors, and in doing so does update_3dpatch_radiation. In this step it calles the TwoStreamMLPMod Solve method. This method fails on a mismatch in the forward solver. This seems to be pretty widespread (a bunch of threads return the same error for various pfts)

General bug information

Typical stack-trace:

 Unknown error submitted to shr_abort_abort.
1375: Image              PC                Routine            Line        Source             
1375: cesm.exe           00000000037D90E7  Unknown               Unknown  Unknown
1375: cesm.exe           000000000351FA62  shr_abort_mod_mp_         110  shr_abort_mod.F90
1375: cesm.exe           0000000002DFDF09  twostreammlpemod_         258  TwoStreamMLPEMod.F90
1375: cesm.exe           0000000002DFBF2E  twostreammlpemod_        1587  TwoStreamMLPEMod.F90
1375: cesm.exe           0000000002AB7AE6  fatesrestartinter        4213  FatesRestartInterfaceMod.F90
1375: cesm.exe           00000000025C59C1  clmfatesinterface        2120  clmfates_interfaceMod.F90
1375: cesm.exe           000000000259F01A  clm_instmod_mp_cl         620  clm_instMod.F90
1375: cesm.exe           00000000026FBA9D  restfilemod_mp_re         215  restFileMod.F90
1375: cesm.exe           000000000259B1BC  clm_initializemod         546  clm_initializeMod.F90
1375: cesm.exe           0000000002524E5B  lnd_comp_nuopc_mp         677  lnd_comp_nuopc.F90
1375: libesmf.so         0000149517B68F95  _ZN5ESMCI6FTable1     Unknown  Unknown
1375: libesmf.so         0000149517B6D86F  ESMCI_FTableCallE     Unknown  Unknown
1375: libesmf.so         000014951827D1CA  _ZN5ESMCI3VMK5ent     Unknown  Unknown
1375: libesmf.so         000014951829D63F  _ZN5ESMCI2VM5ente     Unknown  Unknown
1375: libesmf.so         0000149517B6AA4A  c_esmc_ftablecall     Unknown  Unknown
1375: libesmf.so         000014951853B5E0  esmf_compmod_mp_e     Unknown  Unknown
1375: libesmf.so         00001495187A9D81  esmf_gridcompmod_     Unknown  Unknown
1375: libesmf.so         0000149518DDEA00  nuopc_driver_mp_l     Unknown  Unknown
1375: libesmf.so         0000149518DFBAAE  nuopc_driver_mp_i     Unknown  Unknown
1375: libesmf.so         0000149517B68F95  _ZN5ESMCI6FTable1     Unknown  Unknown
1375: libesmf.so         0000149517B6D86F  ESMCI_FTableCallE     Unknown  Unknown
1375: libesmf.so         000014951827D1CA  _ZN5ESMCI3VMK5ent     Unknown  Unknown
1375: libesmf.so         000014951829D63F  _ZN5ESMCI2VM5ente     Unknown  Unknown
1375: libesmf.so         0000149517B6AA4A  c_esmc_ftablecall     Unknown  Unknown
1375: libesmf.so         000014951853B5E0  esmf_compmod_mp_e     Unknown  Unknown
1375: libesmf.so         00001495187A9D81  esmf_gridcompmod_     Unknown  Unknown
1375: libesmf.so         0000149518DDEA00  nuopc_driver_mp_l     Unknown  Unknown
1375: libesmf.so         0000149518DFBBC4  nuopc_driver_mp_i     Unknown  Unknown
1375: libesmf.so         0000149518E11401  nuopc_driver_mp_i     Unknown  Unknown
1375: libesmf.so         0000149517B68F95  _ZN5ESMCI6FTable1     Unknown  Unknown
1375: libesmf.so         0000149517B6D86F  ESMCI_FTableCallE     Unknown  Unknown
1375: libesmf.so         000014951827D1CA  _ZN5ESMCI3VMK5ent     Unknown  Unknown
1375: libesmf.so         000014951829D63F  _ZN5ESMCI2VM5ente     Unknown  Unknown
1375: libesmf.so         0000149517B6AA4A  c_esmc_ftablecall     Unknown  Unknown
1375: libesmf.so         000014951853B5E0  esmf_compmod_mp_e     Unknown  Unknown
1375: libesmf.so         00001495187A9D81  esmf_gridcompmod_     Unknown  Unknown
1375: cesm.exe           0000000000439E2C  MAIN__                    128  esmApp.F90
1375: cesm.exe           00000000004286A2  Unknown               Unknown  Unknown
1375: libc.so.6          0000149516A3FEB0  Unknown               Unknown  Unknown
1375: libc.so.6          0000149516A3FF60  __libc_start_main     Unknown  Unknown
1375: cesm.exe           00000000004285A5  Unknown               Unknown  Unknown
1375:  ESMF_Finalize: Error closing trace stream

Typical output from where the error happens:

1375: Poor forward solution on two-stream solver
1375:  isol (1=beam or 2=diff):            1
1375:  i (equation):            6
1375:  band index (1=vis,2=nir):            1
1375:  error (tau(i) - omega(i,:)*lambda(:))   -21.7470430886848     
1375:  Dumping Two-stream elements for band    

Followed by multiple blocks like this:

1375:  --           3           3 --
1375:  pft:           0
1375:  area:   0.106609674796023     
1375:  lai,sai:   0.000000000000000E+000  0.000000000000000E+000
1375:  Kb:   0.500000000000000     
1375:  Kb leaf:   0.500000000000000     
1375:  Kd:   0.500000000000000     
1375:  Rb0:   2.775859938347824E-002
1375:  om:   0.500000000000000     
1375:  betad:   0.500000000000000     
1375:  betab:  0.500000000000000     
1375:  a:   0.353553390593274     
1375:  RDiff Down @ bottom:  -2.775859938347825E-002
1375:  RDiff Up @ bottom:   0.000000000000000E+000
1375:  Rbeam @ bottom:   2.775859938347824E-002
1375:  ENDRUN:
1375:  ERROR in TwoStreamMLPEMod.F90 at line 1587     

and finally:

1375:  ENDRUN:
1375:  ERROR in TwoStreamMLPEMod.F90 at line 1587  

This is all repeated from multiple threads.

CTSM version you are using: [output of git describe]

ctsm5.4.002_noresm_v3
sci.1.88.6_api.42.0.0_nor_sci2_api1

Does this bug cause significantly incorrect results in the model's science? [Yes / No]

Hard to tell, there is a crash

Configurations affected: [Fill this in if known.]

Details of bug

See above

Important details of your setup / configuration so we can reproduce the bug

This is not pervasive, but happens for a particular parameter set and overall setup

Important output or errors that show the problem

[maritsandstad]

@rosiealice @JessicaNeedham @mvdebolskiy : This is happening in one specific of Herman's ppe members. I can provide more details on case folder etc here:

/cluster/work/users/herfugl/noresm/ensemble_member.017/run.017/

[maritsandstad]

Just to clarify, the error blocks are from various pfts. The one cited above is from pft number 1, which @JessicaNeedham tells me is maybe bare ground, so it is expected to have 0 sai and 0 lai. Another one from a different pft is this one:

1324:  --           1           2 --
1324:  pft:           4
1324:  area:   5.564352601479575E-002
1324:  lai,sai:    4.97963271362856       0.521094860422489     
1324:  Kb:   0.672231879064447     
1324:  Kb leaf:   0.637932505385259     
1324:  Kd:   0.893998391485858     
1324:  Rb0:    1.00000000000000     
1324:  om:   0.173884011522059     
1324:  betad:   0.581494483470176     
1324:  betab:  0.506273693940217     
1324:  a:   0.823997442838682     
1324:  RDiff Down @ bottom:   1.003751777715042E-002
1324:  RDiff Up @ bottom:   1.581238841293988E-003
1324:  Rbeam @ bottom:   2.477885938935127E-002

The pfts that fail seem to be generally in the lower numbers, but I suspect maybe we are going pft by pft, and hence crashing from the bottom...

[maritsandstad]

@mvertens : Tagging you here to add you to the discussion. Do you have similar fates restart errors, or different fates restart errors? (Or no fates restart errors, that would of course be preferable)

[maritsandstad]

I spoke a bit too soon, possibly @mvertens, I guess the other issue is this one NorESMhub/NorESM#770 . However, I thought this one might possibly be related (it least it is not totally impossible)? As far as I can tell, and after some discussion via email with Ryan Knox, this one is due to a radiation forward solver code not converging for the restart. If the restart problem is sufficiently far from being bit-for-bit in the initiallisation, I suppose that non-convergence could happen from that? (May also be totally unrelated...)

[mvertens]

@maritsandstad - this is the issue that both @mvdebolskiy and myself are currently trying to sort out. NorESMhub/NorESM#770 (comment). If this is related that would be great. I still suspect that the restart problem is related to how FATES is being initialize - but of course it's very hard to prove this. Is there something we can change to possible work around this in a test case (e.g. increase the number of iterations for convergence, etc.)

[maritsandstad]

I think if they are related, I think it is related in the way, that this occurence of a restart crash because the forward radiation solver is caused by an erroneous FATES initialisation, which is bad enough, that the radiation solver cannot converge (in this particular case). My suspicion for this particular issue is that the expected read in input is somehow at a mismatch which makes the radiation solver convergence infeasible, and if the initialisation at restart is faulty, that could be doing that only in this case, the consequences are more extreme (crash and doesn't run)... That may be overall unhelpful, though, @mvertens

[JessicaNeedham]

FYI one of my runs just crashed with this same error message:

Poor forward solution on two-stream solver
380: isol (1=beam or 2=diff): 1
380: i (equation): 6
380: band index (1=vis,2=nir): 1
380: error (tau(i) - omega(i,:)*lambda(:)) 3.436898822873347E-004
380: Dumping Two-stream elements for band 1
380:
380: rbeam atm: 1.00000000000000
380: rdiff_atm: 1.00000000000000
380: alb grnd diff: 8.000000000000000E-002
380: alb grnd beam: 8.000000000000000E-002
380: cosz: 0.640349689469494
380: snow fraction: 0.000000000000000E+000
380: -- 1 1 --
380: pft: 1
380: area: 1.00000000000000
380: lai,sai: 3.71828399405890 0.371828399559679
380: Kb: 0.842078586951975
380: Kb leaf: 0.826286445640641
380: Kd: 1.03519339863254
380: Rb0: 1.00000000000000
380: om: 0.173727272728674
380: betad: 0.580805965469270
380: betab: 0.468468047471283
380: a: 0.954104398321617
380: RDiff Down @ bottom: 2.336969402545246E-002
380: RDiff Up @ bottom: 2.569158538803989E-003
380: Rbeam @ bottom: 3.193042171386475E-002

and many similar blocks.

I am my own branch https://github.com/JessicaNeedham/fates/tree/ncsrevise-combo which is up to date with noresm but with changes from various pending PRs added. The case is on betzy here: /cluster/work/users/jessica/ncsrevise_runs/noresm-fates-ne16-LU-PPE-AD-spinup.N2fbd5fff8-Fab73a7eb7.2026-02-26. I did not have the recent restart fix from NorESMhub/fates#54 yet, so maybe I should try again with that included?