Crashes when attempting to run with custom high-resolution ERA5 Land meteorological forcing

[korsbakken]

Hi @mvdebolskiy. @kjetilaas and @rosiealice. Here's an update on the high-resolution regional NorSink grid and crashes with the ERA5 Land forcing data, as discussed in the CTSM Friday meeting on January 23. I posted it in the Teams meeting hat, but not sure that everybody gets notified or has access, so moving it here. Let me know if you prefer some other channel.

I have tested several more things since the meeting. There are still some avenues to try, but great if you could read the questions at the end @mvdebolskiy and reply to those if you know. Sorry about the long post, there are lots of possibly relevant details.

Path to the case and repo directory are in the first comment.

To recap: I have produced a mesh for a minimal rectangular 0.1x0.1 degree grid that covers Norway and all river outflows, and a surface data set for the same mesh. I have also processed ERA5 Land meteorological data into a 3-stream forcing data set on the same grid and 1-hourly resolution. I converted them to the same format as CRUNCEP, CRUJRA, etc. Finally, I added the new mesh, surface data files and streaming data fles to the XML database files so that I can use them with create_newcase and the CIME scripts.

The modifications are in a fork under my own user, in the branch norsink_inputdata_main, here: https://github.com/korsbakken/CTSM/tree/norsink_inputdata_main. I have put a clone set to the right commit in my work folder on betzy (see first comment).

The modifications are in XML files in CTSM, ccs_config and cdeps, and in some files in the mksurfdat pipeline. I don't think I have changed anything in code that executes during model runs.

I have created, built and successfully run cases with the new mesh and surface data and standard CRUJRA forcing data (CRUJRA2024). But when I swap in the ERA5 Land-based forcing data I created, the model crashes after about 40 seconds, with no discernable error messages in the logs, other than cesm.log stating that MPI_ABORT was invoked in MPI_COMM_WORLD with error code -1, no indication of which component or function caused the abort. In the stack traces, the only discernable function name (at the third stack level) is PIOc_read_darray+0x41b. The rest are just hex codes.

The other log files don't indicate much activity beyond the initialization before they just stop, with no error messages. The only exception is the atm.log file which does a lot of initialization activities, and then states that it opens the first forcing file for the first time step, and attempts to read FSDS. It doesn't appear to finish. You can see the details in the run subdirectory in the case directory.

I have experimented with adding some debug writes to cdeps/streams/dshr_strdata_readstrm.F90 to pinpoint exactly where the crash happens, but that's very slow going. I haven't found a smoking gun yet, and I can't be totally sure that the actual crash doesn't happen in a completely different component that is executing in parallel.

Of the things I haven't corrected or tested yet, there are two main candidates for the cause, and two less likely ones. @mvdebolskiy, if you know, could you weigh in on whether any of these would produce silent crashes of the type I describe above?:

1. The ERA5 Land data contains missing values at some points that are not masked out, and for some points it also depends on time whether they have missing values or not. I plan to test filling these with nearest-neighbor values to see what happens.

2. The processed ERA5 Land files end up using NaN for fill values and missing values, whereas the standard data sets appear to use big float values like 1e36 instead. Which means that the missing values above also end up as NaNs. I don't know whether the model requires a non-NaN float as a fill value, but I can try to make the processing code use 1e36 instead.

3. The processed files record time as hours from the start of the month, while the standard datasets use days. The metadata in the files indicate this in a way that should be compliant with CF standards, and they get parsed correctly by the xarray and netCDF4 libraries in Python at least. But does the model require using fractional days instead?

4. The processed ERA5 Land files use proleptic_gregorian calendar, not noleap. As far as I understand, DATM will just ignore February 29 in this case, and there's no leap year in the test runs I have done anyway. But I can test converting the data to noleap if having different calendar IDs in the input files is in fact an issue.

[korsbakken]

You can look at this case directory on betzy: /cluster/work/users/janko/noresm/test_NorwayRect_with_era5land_forcing_REF/

Note that the original directory name did not have _REF at the end, so the nane may differ in logs and replay scripts. I have backed up a ton of case directories where I have tested things, and just renamed them with a suffix before creating new active cases with the same unsuffixed directory name.

A clone of the repo at the right commit is here: /cluster/work/users/janko/repos/ctsm_era5land_testing/CTSM/

[mvertens]

@korsbakken - can you please point me to your run directory and make it readable. There is a way to translate hex files to line of code. If you are in your run directory - you can do the following:
addr2line -e ../bld/cesm.exe <hex code - eg. 0x48f417>
It helps to first isolate all of the output from a single MPI task in the cesm.log file (the task prefix accompanies every line of output) - and then tracking down the component where the model is crashing. If it is dying in DATM - it's also possible run just DATM in ERA5 mode without CLM to debug the problem there. I'm happy to help with this if that would be useful - since I've been in the process of refactoring CDEPS and ERA5 is the one outstanding mode that is not routinely tested.

[mvertens]

The run is crashing in datm. Looking at /cluster/work/users/janko/noresm/test_NorwayRect_with_era5land_forcing/run

grep 100: cesm.log.1414463.260126-160000
gives the following:

100: /cluster/work/users/janko/noresm/test_NorwayRect_with_era5land_forcing/bld/cesm.exe() [0x4b0cd19]
100: /cluster/work/users/janko/noresm/test_NorwayRect_with_era5land_forcing/bld/cesm.exe() [0x4b0cccb]
100: /cluster/work/users/janko/noresm/test_NorwayRect_with_era5land_forcing/bld/cesm.exe(PIOc_read_darray+0x41b) [0x4b3b49b]
100: /cluster/work/users/janko/noresm/test_NorwayRect_with_era5land_forcing/bld/cesm.exe() [0x4b00817]
100: /cluster/work/users/janko/noresm/test_NorwayRect_with_era5land_forcing/bld/cesm.exe() [0x4b006b1]
100: /cluster/work/users/janko/noresm/test_NorwayRect_with_era5land_forcing/bld/cesm.exe() [0x4848a3b]
100: /cluster/work/users/janko/noresm/test_NorwayRect_with_era5land_forcing/bld/cesm.exe() [0x483ae6b]
100: /cluster/work/users/janko/noresm/test_NorwayRect_with_era5land_forcing/bld/cesm.exe() [0x4812296]
100: /cluster/work/users/janko/noresm/test_NorwayRect_with_era5land_forcing/bld/cesm.exe() [0x9043c3]
100: /cluster/work/users/janko/noresm/test_NorwayRect_with_era5land_forcing/bld/cesm.exe() [0x90210f]
100: --------------------------------------------------------------------------
100: MPI_ABORT was invoked on rank 100 in communicator MPI_COMM_WORLD
100: with errorcode -1.
100:
100: NOTE: invoking MPI_ABORT causes Open MPI to kill all MPI processes.
100: You may or may not see output from other processes, depending on
100: exactly when Open MPI kills them.
100: --------------------------------------------------------------------------
[/cluster/work/users/janko/noresm/test_NorwayRect_with_era5land_forcing/run] (test_NorwayRect_with_era5land_forcing *+)
$ addr2line -e ../bld/cesm.exe 0x4848a3b
/cluster/home/janko/src/CESM_repos/CTSM/components/cdeps/streams/dshr_strdata_mod.F90:1761
[/cluster/work/users/janko/noresm/test_NorwayRect_with_era5land_forcing/run] (test_NorwayRect_with_era5land_forcing *+)
$ addr2line -e ../bld/0x483ae6b
addr2line: '../bld/0x483ae6b': No such file
[/cluster/work/users/janko/noresm/test_NorwayRect_with_era5land_forcing/run] (test_NorwayRect_with_era5land_forcing *+)
$ addr2line -e ../bld/cesm.exe 0x483ae6b
/cluster/home/janko/src/CESM_repos/CTSM/components/cdeps/streams/dshr_strdata_mod.F90:1410
[/cluster/work/users/janko/noresm/test_NorwayRect_with_era5land_forcing/run] (test_NorwayRect_with_era5land_forcing *+)
$ addr2line -e ../bld/cesm.exe 0x4812296
/cluster/home/janko/src/CESM_repos/CTSM/components/cdeps/streams/dshr_strdata_mod.F90:970
[/cluster/work/users/janko/noresm/test_NorwayRect_with_era5land_forcing/run] (test_NorwayRect_with_era5land_forcing *+)
$ addr2line -e ../bld/cesm.exe 0x9043c3
/cluster/home/janko/src/CESM_repos/CTSM/components/cdeps/datm/atm_comp_nuopc.F90:674
[/cluster/work/users/janko/noresm/test_NorwayRect_with_era5land_forcing/run] (test_NorwayRect_with_era5land_forcing *+)

I think the problem is that your forcing data is NetCDF4 - and I believe that pio expects cdf5.
Can you try the following for your forcing data:
nccopy -k cdf5 /cluster/shared/noresm/inputdata/cicero_mods/atm/datm7/atm_forcing.datm7.ERA5LAND_NORWAYRECT.0.1d.c20260120/Solar1Hrly/clmforc.ERA5Land_NorwayRect0.1x0.1.Solr.2019-01.nc

[korsbakken]

@korsbakken - can you please point me to your run directory and make it readable. There is a way to translate hex files to line of code. If you are in your run directory - you can do the following: addr2line -e ../bld/cesm.exe <hex code - eg. 0x48f417> It helps to first isolate all of the output from a single MPI task in the cesm.log file (the task prefix accompanies every line of output) - and then tracking down the component where the model is crashing. If it is dying in DATM - it's also possible run just DATM in ERA5 mode without CLM to debug the problem there. I'm happy to help with this if that would be useful - since I've been in the process of refactoring CDEPS and ERA5 is the one outstanding mode that is not routinely tested.

By run directory, I assume you mean /run under the case directory? If so, the case directory is in the first comment, and should already be readable. From your second comment, I assume you were able to access it, but let me know if you meant some other directory.

[mvertens]

So I think I'm not correct about the lack of support for NetCDF4 (I just checked with the PIO author).
By the run directory - I mean /cluster/work/users/janko/noresm/test_NorwayRect_with_era5land_forcing/run
And if you look at atm.log.1414463.260126-160000 - it's dying trying to read in FSDS
(shr_strdata_readstrm) reading FSDS into Faxa_swdn at time index: 1

[korsbakken]

I think the problem is that your forcing data is NetCDF4 - and I believe that pio expects cdf5. Can you try the following for your forcing data: nccopy -k cdf5 /cluster/shared/noresm/inputdata/cicero_mods/atm/datm7/atm_forcing.datm7.ERA5LAND_NORWAYRECT.0.1d.c20260120/Solar1Hrly/clmforc.ERA5Land_NorwayRect0.1x0.1.Solr.2019-01.nc

Thanks @mvertens, I will test that just to be sure. I suspect that's not the issue though. I have made a successful run with CRUJRA forcing data (0.5 degree and 6-hourly resolution, DATM_MODE=CLMCRUJRA2024). Those files are also reported as being netCDF-4 by ncdump -k.

The files for that mode are at /cluster/shared/noresm/inputdata/atm/datm7/atm_forcing.datm7.CRUJRA.0.5d.c20241231/three_stream.

And yes, I saw the last line. But like I wrote in the original post, I just wasn't 100% sure that it wasn't some other part of the model that triggered MPI_ABORT in parallel, and shr_strdata_readstrm got killed by that before it got any further. To my understanding, if any part of the model invokes MPI_ABORT, all the other tasks get hit with a kill -9? But it's good to know the hex codes in the stack trace confirm that the crash happens in the read itself.

[mvertens]

There are several problems with datasets you are using.

• First the time dimension is int64:
  int64 time(time) ;
  time:long_name = "observation time" ;
  time:units = "hours since 2019-01-01 00:00:00" ;
  time:calendar = "proleptic_gregorian" ;
  Could you convert this to int?
• We do not support the calendar proleptic_gregorian in CDEPS? We only support GREGORIAN or NO_LEAP. It's odd that CDEPS did not die with an unmatched calendar.
• The NaN for the Fill_value could be problematic. Could you try 1.e38 - or 1.e35?
• ncview will not display this file and it gives errors of the form

ncview: netcdf_dim_value: unknown data type (10) for
dimension time

[mvertens]

@korsbakken - the problem is that an iodescriptor is not being created. There is a logic problem in CDEPS where it doesn't crash if you have 3 dimensions and you have not created an io descriptor. Can you please try to make your time dimension unlimited in these files and try again. In the meantime we need to fix CDEPS to error out if you do not have an io descriptor created.

[mvertens]

@korsbakken - the problem is not that the time is not an unlimited dimension but that it has the wrong order in the dimension ordering of FSDS.
As an example - here is a file
$ ncdump -h /cluster/shared/noresm/inputdata/atm/datm7/atm_forcing.datm7.Qian.T62.c080727/Solar6Hrly/clmforc.Qian.c2006.T62.Solr.1972-01.nc
that has the following dimensions:

	time = 124 ;
	scalar = 1 ;
	lat = 94 ;

and FSDS:
float FSDS(time, lat, lon)
And this works fine:
However, in your case
/cluster/shared/noresm/inputdata/cicero_mods/atm/datm7/atm_forcing.datm7.ERA5LAND_NORWAYRECT.0.1d.c20260120/Solar1Hrly/clmforc.ERA5Land_NorwayRect0.1x0.1.Solr.2019-01.nc
FSDS has the following ordering of dimensions:
double FSDS(lat, lon, time)
and is problematic.
In the latest CDEPS version of the code you will get an error. I actually merged your CDEPS code in your branch up to the latest NorESMHub version and created your case with just DATM and was able to get the following error:
(shr_strdata_set_stream_iodesc) the third dimension of a 3d field must be either time or a vertical level

[mvdebolskiy]

@korsbakken
I see that your sandbox for CTSM is based on ctsm5.3.045_noresv_v0. First I would recommend you to pull/merge HEAD from NorESMhub/CTSM:noresm and also update the submodules, probably, you will also need to do it for CDEPS. Other than that, you can see Mariana's comments above, though int64 is a valid netcdf data type, I am not sure when exactly it has been added, so some versions of netCDF might not support it. In addition, in general, all stream data files for CESM/NorESM usually have an unlimited time dimension. Thirdly, you will run into calendar problems with the ERA5 forcing you are using right now. I would recommend to use cdo/nco to change the calendar to gregorian/no_leap.

[korsbakken]

@korsbakken - the problem is not that the time is not an unlimited dimension but that it has the wrong order in the dimension ordering of FSDS.

Thanks @mvertens. That was indeed the cause of this particular crash. I made a version of the forcing data where I only changed the dimension ordering, and it then managed to read the files and progress quite a bit, before crashing on something else.

That was just a test. I will try to fix the fill value and missing value issues next, and change the time variable to be a float in units of "days since" instead of "hours since", in case that's also an issue. And maybe change the data to be float32 instead of double/float64 across the board, like it is in the CRUJRA files. I don't know yet whether using float64 causes issues, but in any case there's no reason to take up that extra space. It looks like the numbers in the original ERA5 Land grib files are single-precision anyway.

After putting the time dimension first, the model now crashes with the message forrtl: error (75): floating point exception in one of the calls on line cdeps/streams/dshr_methods_mod.F90 line 132, which is the last line in the following if statement:

             if (size(dataPtr1d) > 0) then
                write(msgString,'(A,3g14.7,i8)') trim(string)//': '//trim(lfieldnamelist(n)), &
                     minval(dataPtr1d), maxval(dataPtr1d), sum(dataPtr1d), size(dataPtr1d)

It's not clear exactly which variable is being checked when the crash happens. But if it's naively feeding an array of NaNs into the sum function here, I assume that might produce a "floating point exception" of some sort.

Further up in cesm.log, it says: Caught signal 8 (Floating point exception: floating-point overflow). I don't know how the sum function above is implemented. But the binary representation of a NaN float involves setting all the bits in the exponent to 1, so if you naively add two such numbers, you would almost certainly get an overflow.

If you are curious, the case directory is /cluster/work/users/janko/noresm/test_NorwayRect_with_era5land_forcing_REORDERED_DIMS. But it's probably not worth spending time on it until I've fixed the fill values and missing values, and tested with that.

[korsbakken]

in general, all stream data files for CESM/NorESM usually have an unlimited time dimension. Thirdly, you will run into calendar problems with the ERA5 forcing you are using right now. I would recommend to use cdo/nco to change the calendar to gregorian/no_leap.

Thanks @mvdebolskiy. I will try that after testing with a sensible _FillValue and filling in the missing values that shouldn't be missing. I assume the fill values / missing values in the forcing data in grid cells that aren't masked out will cause problems?

About gregorian vs proleptic_gregorian: In the CF conventions, gregorian (not proleptic) or standard means the historical Gregorian calendar, i.e., Julian leap year rules before 1582, Gregproam rules after 1582, and 1582-10-04 is followed directly by 1582-10-15, with 1582-10-05 through 1582-10-14 being non-existent. Is that supported, but proleptic_gregorian (Gregorian rules extrapolated backwards before 1582 and no missing dates) is not? Actually, in the latest version of the standard, it looks like the gregorian identifier isn't supported at all, only standard and proleptic_gregorian (table 4.1 of CF Conventions v1.13).

[mvdebolskiy]

@korsbakken I know how CF conventions work. However, the model might give you an error if it reads calendar attribute and tries to compare ti with the one available in the model and through you an error. I remember encountering this in some earlier verison, maybe it us gone, but in any case. If you are doing spinup, you can not run it with gregorian.
ERA5 used to have gregorian calendar attribute on the files directly from copernicus and there was no issue, however, that changed few years ago.

[korsbakken]

If you are doing spinup, you can not run it with gregorian.

Yes, that's a good point, thanks. I think Iris mentioned she had some issues with the spinup related to February 29. According to the CDEPS docs, DATM should handle different calendars in the stream files and in the model automatically, and skip Feb 29 as needed. But that might not apply during a spinup (or at all if NorESM is on a different version than the latest master?).

[mvertens]

@korsbakken - thanks for the update and for pointing out your latest caseroot. I know where to put a print statement regarding the floating point overflow - so I'm happy to try that tomorrow.