"floating overflow" error in LIS

[bhanu-magotra]

Hi,
I am running LISF 7.4 with NoahMP3.6 and HyMAP2 in "restart" mode. I am using ESMWF's ERA5 with MERRA2 in overlay at 1 hour model timestep with outputs at daily timestep.
Earlier this week, I faced a "positive ERRWAT" error, which I resolved by increasing the NoahMP3.6's water balance check threshold to 10.

However, now I am facing a new "floating overflow" error which is recorded in the PBS error log as below :

application called MPI_Abort(comm=0x84000002, 1) - process 10
application called MPI_Abort(comm=0x84000002, 1) - process 11
real 49814.66
user 0.01
sys 0.07
forrtl: error (72): floating overflow
Image              PC                Routine            Line        Source             
LIS                00000000014258A4  Unknown               Unknown  Unknown
libpthread-2.17.s  00002B447284D630  Unknown               Unknown  Unknown
LIS                00000000004BC4E0  hymap2_modelmod_m         393  HYMAP2_modelMod.F90
LIS                00000000004C17AE  hymap2_model_core         402  HYMAP2_model_core.F90
LIS                00000000004BB33A  hymap2_model_             271  HYMAP2_model.F90
LIS                000000000050FB58  hymap2_routing_ru         563  HYMAP2_routing_run.F90
LIS                00000000006F9A6E  lis_routingmod_mp         303  LIS_routingMod.F90
LIS                000000000129738B  retrospective_run         208  retrospective_runMod.F90
LIS                0000000000D36EA9  MAIN__                     70  lisdrv.F90
LIS                0000000000416362  Unknown               Unknown  Unknown
libc-2.17.so       00002B4472D7E555  __libc_start_main     Unknown  Unknown
LIS                0000000000416269  Unknown               Unknown  Unknown
real 30113.93
user 319831.58

The PBS output file logs the following :

csky067.hpc.iitd.ac.in,csky117.hpc.iitd.ac.in
csky067: Attaching 18210 to 3249834.pbshpc
csky117: Attaching 37152 to 3249834.pbshpc
Attaching 37217 to 3249834.pbshpc

===================================================================================
=   BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES
=   PID 18298 RUNNING AT csky067.hpc.iitd.ac.in
=   EXIT CODE: 6
=   CLEANING UP REMAINING PROCESSES
=   YOU CAN IGNORE THE BELOW CLEANUP MESSAGES
===================================================================================
   Intel(R) MPI Library troubleshooting guide:
      https://software.intel.com/node/561764
===================================================================================

The lislog.xxxx does not log anything specific to the error, and just ends while processing 19870920 timestep :

INFO] LIS cycle time: 09/19/1987 21:00:00
[INFO] LIS cycle time: 09/19/1987 22:00:00
[INFO] LIS cycle time: 09/19/1987 23:00:00
[INFO] LIS cycle time: 09/20/1987 00:00:00
 [INFO] Writing surface model output to:  
 ./output/LIS_ERA5/SURFACEMODEL/198709/LIS_HIST_198709200000.d01.nc
 [INFO] Writing routing model output to:  
 ./output/LIS_ERA5/ROUTING/198709/LIS_HIST_198709200000.d01.nc
[INFO] LIS cycle time: 09/20/1987 01:00:00
 [INFO] Reading MERRA-2 file (bookend,           2  ... 
 ../../datasets/MERRA2/MERRA2_100/Y1987/M09/MERRA2_100.tavg1_2d_slv_Nx.19870921.
 nc4
 [INFO] Reading MERRA-2 file (bookend,           2  ... 
 ../../datasets/MERRA2/MERRA2_100/Y1987/M09/MERRA2_100.tavg1_2d_flx_Nx.19870921.
 nc4 (for lowest model level fields)
 [INFO] Reading MERRA-2 file (bookend,           2  ... 
 ../../datasets/MERRA2/MERRA2_100/Y1987/M09/MERRA2_100.tavg1_2d_flx_Nx.19870921.
 nc4 (for corrected precipitation fields)
 [INFO] Reading MERRA-2 file (bookend,           2  ... 
 ../../datasets/MERRA2/MERRA2_100/Y1987/M09/MERRA2_100.tavg1_2d_rad_Nx.19870921.
 nc4
[INFO] LIS cycle time: 09/20/1987 02:00:00
[INFO] LIS cycle time: 09/20/1987 03:00:00
[INFO] LIS cycle time: 09/20/1987 04:00:00
[INFO] LIS cycle time: 09/20/1987 05:00:00
[INFO] LIS cycle time: 09/20/1987 06:00:00
[INFO] LIS cycle time: 09/20/1987 07:00:00
[INFO] LIS cycle time: 09/20/1987 08:00:00
[INFO] LIS cycle time: 09/20/1987 09:00:00
[INFO] LIS cycle time: 09/20/1987 10:00:00
[INFO] LIS cycle time: 09/20/1987 11:00:00
[INFO] LIS cycle time: 09/20/1987 12:00:00
[INFO] LIS cycle time: 09/20/1987 13:00:00
[INFO] LIS cycle time: 09/20/1987 14:00:00
[INFO] LIS cycle time: 09/20/1987 15:00:00
[INFO] LIS cycle time: 09/20/1987 16:00:00
[INFO] LIS cycle time: 09/20/1987 17:00:00

The HyMAP2_modelMod.F90 contains the following code at line 393:

LISF/lis/routing/HYMAP2_router/HYMAP2_modelMod.F90

Line 393 in 9d9c5fe

(1.+grv*dt*manval**2.*abs(dout_pre)*dflw_imp**(-7./3))

I would appreciate if someone could help me resolve this issue. I am attaching the lisconfig file for reference.

lis_config.zip

Thank you.

[agetiran]

Hi @bhanu-magotra,
I see in your lis.config file that you are using "1hr" time step for both HyMAP and Noah. I can also see that you are using "local inertia" as routing method. As opposed to the "kinematic" wave option, local inertia may result in numerical instability and crash. Numerical stability is reached by using short time steps. The "adaptive" time step option is built for that purpose. That option breaks the user defined time step into optimal time steps. Optimality is based on the alfa coefficient. Ideally, that coefficient should be lower than 1.

All that said, I suggest that you update the following options in your lis.config file:

HYMAP2 routing model time step: "15mn"
HYMAP2 routing model adaptive time step alfa coefficient: 0.5
Noah-MP.3.6 model timestep: "15mn"

These options will result in longer, but more stable runs. I also suggest that you don't restart HyMAP from a previous run using the previous configuration. Instead, you should "coldstart" the model with the new configuration. If the model keeps crashing at the same line, you can play with the alfa coefficient.

Alternatively, if local inertia isn't really needed in your simulations, you can keep your original configuration and simply use "kinematic", as follows:

HYMAP2 routing method: "kinematic"

That will insure stability in your runs.

I hope this helps.

Augusto

[bhanu-magotra]

Thank you @agetiran for valuable help. The runs were successfully completed using your suggestions.