Issue in running Noah-MP 4.0.1 Open-loop experiment

[tais958]

Hi everyone. I was trying to repeat OL experiment similar to testcase2 but replacing Noah 3.6 model with Noah-MP 4.0.1. It went smoothly when the model domain and resolution are unchanged. However, after using a domain with ~400 x 400 grids (dx=0.01, dy=0.01), it stopped running when it got to the point of output netcdf file. (I have tested using different output frequency and every time it stopped right at the first time it should output)

Below are the sbatch script, lislog, lis.config, and lis.err:
script.lis.txt
lislog_ol.0000.txt
lis.config_noahmp401_ol.txt
lis.err.txt

Any comment or suggestion is much appreciated!

[dmocko]

Hi @tais958

Can you share your MODEL_OUTPUT_TBL file? Some variables are not available in Noah-MP-4.0.1 to be output, that may be available in Noah-3.6. This could be why it stops running only when you try to write output.

Can you also share your ldt.config file? Finally, look for other lislog_ol.????.txt files, particularly for any that have a different message at the end of the file, or are larger than the others. If there are any, please share as well.

From looking at your lis.err.txt file, however, I am seeing MPI errors, and I am not sure why you are seeing those as well.

[dmocko]

HI @tais958 - can you please provide the ldt.config and lis.config files for both the 0.1-deg domain (that works for you) as well as these two files for a finer grid space (that you are saying does not work)? Also, please provide the lislog.0000 file and any standard output error messages for the domain that is not working for you.

[tais958]

Hi @dmocko, please find the files in the attached zip file. Let me know if you need more info. Thanks!
Archive.zip

[tais958]

Hi @dmocko, have you had a chance to look at the files? Please let me know if they work for your need. Thanks.

[dmocko]

@tais958 Yes, but I would like more info please.

Do you submit this as a job, or run it on the command line? When it crashes (either to the job standard output/error, or to the screen), is there any error message printed?

Also, can you please check the size of all lislog files? It may be that one (or more) of the processors is giving an error, so look for lislog file(s) that are larger than the others. Look particularly at the end of these files. You can also try "tail lislog_ol*" to see if any error messages are at the end of any of the lislog files.

[tais958]

@dmocko Thanks for the reply.

I submitted as a job. The error was printed and showing the crash was due to the calculation in module_sf_noahmplsm_401.F90 was divided by zero as what I provided above. I just tried using OPT_BTR = 2 and it successfully bypass where it was crashed, but another error came up in different line of code. Based on the tests, I guess it may not be the memory issue.

I've also checked all the log files. It usually stopped after [INFO] Resetting tair_agl_min for RHMin calculation
, and before [INFO] Writing surface model output to:

I was thinking maybe something in the step of LSM parameter processing that I could pay attention to? Especially those options may be more resolution-sensitive? Any comments?

Thanks!

[dmocko]

Hi @tais958

Thanks for your patience while we have looked into this. Thanks for sharing your full ldt and lis config files. I tested them on my end, and reproduced your error.

The crash is happening at a point where the land cover is urban, but the soil texture is considered to be water. Based on the SOILPARM look-up table, the parameter values for the "water" soil texture type will cause these crashes.

Is there any reason that you need to run with the "Openwater" surface model type? If not, you should turn it off in your ldt.config as well as in your lis.config files:

Number of surface model types:          1
Surface model types:                    "LSM"

One other thing that should be changed in your ldt.config is that you need a "/" between "INPUT" and "LS_PARAMETERS" on this line:

Aspect map: ./INPUTLS_PARAMETERS/topo_parms/SRTM30/raw_wgtopo30antarc

If you turn off "Openwater", the model will run as expected, as you won't have any land points with the "water" soil texture type.

If you do need "Openwater", please let us know, and we can see if there's another solution.

---

Relatedly, I noticed this config entry in your ldt.config file:

Soil texture force exclusion of water points during fill: true

I wasn't familiar with this config option, but changing it to ".true." or to ".false." did not change the model crashes without also removing the "Openwater" surface model type.

@emkemp - I see in the code that you added this config entry; however, there is not any description of this option in the LDT Users' Guide. When you are able, can you please add it? Also, is this option designed at all to prevent the above inconsistency from happening?

[tais958]

Hi @dmocko, thanks so much for looking into this issue for me! I followed what you suggested and it did solve the problem. I had no idea if I should turn off "Openwater" surface model as I basically borrowed the config file from the testcase. While it took me fair amount of time to figure out what happened, I learned something from the process. Thank you!

[emkemp]

@dmocko Thanks for pointing this out. I just submitted a pull request to fix.