Unable to read in CRTM coefficients, using inline post, while running GNU executables on Hera

[MichaelLueken]

Description

While running the weather model on Hera using GNU-built executables, inline post is failing while using the postxconfig-NT-rrfs.txt post config file. This configuration uses simulated radiances, thus requiring to read in CRTM coefficients. The inline post is failing with the following error messages:

 Check_Binary_File(FAILURE) : Data file needs to be byte-swapped.
 Open_Binary_File(FAILURE) : Error checking imgr_g15.SpcCoeff.bin file byte order
 SpcCoeff_ReadFile(Binary)(FAILURE) : Error opening imgr_g15.SpcCoeff.bin
 CRTM_SpcCoeff_Load(FAILURE) : Error reading SpcCoeff file #1, imgr_g15.SpcCoeff.bin; Process ID: 0
 CRTM_Init(FAILURE) : Error loading SpcCoeff data; Process ID: 0
 ERROR*** crtm_init error_status=      3

Looking in cmake/GNU.cmake, I noted that there was no -fconvert=big-endian in the Fortran flags. Adding this flag allows the inline post to successfully run, but other tests failed in the weather model:

Error termination. Backtrace:
At line 4448 of file /scratch1/NCEPDEV/stmp2/Michael.Lueken/ufs-srweather-app/sorc/ufs-weather-model/FV3/ccpp/physics/physics/MP/Thompson/module_mp_thompson.F90 (unit = 63, file = 'qr_acr_qgV2.dat')
Fortran runtime error: End of file

Should inline post not be run using GNU-built executables?

To Reproduce:

The SRW App only runs GNU tests on Hera, but this would likely apply to all machines that can allow for GNU builds.

1. Clone my feature/hash_update branch on Hera - git clone -b feature/hash_update [email protected]:MichaelLueken/ufs-srweather-app.git
2. cd ufs-srweather-app
3. Checkout externals - ./manage_externals/checkout_externals
4. Build the GNU executables - ./devbuild.sh -p=hera -c=gnu
5. module use modulefiles
6. module load wflow_hera
7. conda activate srw_app
8. cd tests/WE2E
9. Run the fundamental WE2E test suite using ./run_WE2E_tests.py -t fundamental -m hera -a <account> -c gnu (replace with your project on Hera) and see the inline post test fail with the above error message in ../../../expt_dirs/grid_RRFS_CONUScompact_25km_ics_HRRR_lbcs_RAP_suite_RRFS_v1beta/log/run_fcst_mem000_2020081000.log

[DusanJovic-NOAA]

Inline post should run using the GNU compiler.

Since the code currently does not have -fconvert flag set at the top level, the executable will not convert fortran unformatted files to big-endian. I assume 'qr_acr_qgV2.dat' file is little-endian. Otherwise, code will fail while reading it.

We should use all fortran unformatted files using native endianness, which is little endian these days on all the platforms we support, and avoid setting the conversion flag globally.

I see CRTM provides both little and big endian version of this file:

https://github.com/JCSDA/crtm/blob/v2.4.0_emc.3/fix/SpcCoeff/Little_Endian/imgr_g15.SpcCoeff.bin

https://github.com/JCSDA/crtm/blob/v2.4.0_emc.3/fix/SpcCoeff/Big_Endian/imgr_g15.SpcCoeff.bin

Can you try to use little endian version?