Fix 06z gfs_sfcanl jobs failing due to a bug in CCPP

[DavidHuber-NOAA]

Description

This fixes a bug in CCPP that prevented the 06Z gfs_sfcanl job from running global_cycle on the 03Z IAU time. The code change now allows the input hour to be any integer between 0 and 23.
Resolves #4364
Refs #4408 (partially resolves but a full investigation is needed)

Type of change

• Bug fix (fixes something broken)
• New feature (adds functionality)
• Maintenance (code refactor, clean-up, new CI test, etc.)

Change characteristics

• Is this change expected to change outputs YES: the CCPP code change itself is non-answer-changing, but it requires a model hash update
  • GFS
  • GEFS
  • SFS
  • GCAFS
• Is this a breaking change (a change in existing functionality)? NO
• Does this change require a documentation update? NO
• Does this change require an update to any of the following submodules? YES
  • UFS-utils Update ccpp-physics to enable non 6-hourly global cycle updates ufs-community/UFS_UTILS#1122 (depends on Allow ihr to be non-multiples of 6 ufs-community/ccpp-physics#336)
  • UFS-weather-model Allow ihr to be non-mulitples of 6 in ccpp-physics ufs-community/ufs-weather-model#3036 (comment) (depends on Allow ihr to be non-multiples of 6 ufs-community/ccpp-physics#336)

How has this been tested?

• C96_atm3DVar_extended test on WCOSS2
• UFS_Utils regression tests on Ursa (no change to baseline)
• UFS regression tests
• Full suite of GW tests on all platforms (when the UFS model hash is ready)

Checklist

• Any dependent changes have been merged and published
• My code follows the style guidelines of this project
• I have performed a self-review of my own code
• I have commented my code, particularly in hard-to-understand areas
• I have documented my code, including function, input, and output descriptions
• My changes generate no new warnings
• New and existing tests pass with my changes
• This change is covered by an existing CI test or a new one has been added

[aerorahul]

Suggested change

	local MOM6_OUTPUT_DIR="${MOM6_OUTPUT_DIR:-./MOM6_OUTPUT}"
	local MOM6_RESTART_DIR="${MOM6_RESTART_DIR:-./MOM6_RESTART}"
	local MOM6_OUTPUT_DIR="./MOM6_OUTPUT"
	local MOM6_RESTART_DIR="./MOM6_RESTART"

This isn't an option. The workflow will always write to this space. The other instances where this variable is set in this manner is ush/parsing_namelists_FV3.sh

[JessicaMeixner-NOAA]

I agree, I don't think this should be configurable. This is assumed other places.

I also see that ufs-community/ufs-weather-model@6f22f57...f8b0802 we have a MOM6_OUTPUT_FH defined, which is defined in forecast_predet.

I'm assuming those are consistent definitions?

[aerorahul]

just one suggestion, looks good.

[JessicaMeixner-NOAA]

We should run a S2S test of some sort to make sure ocean output looks okay given the changes. I didn't follow exactly when those changes went in. @jiandewang or @dpsarmie might know more about that the MOM6_OUTPUT_FH to make sure that is as expected here now.

[JessicaMeixner-NOAA]

I agree, I don't think this should be configurable. This is assumed other places.

I also see that ufs-community/ufs-weather-model@6f22f57...f8b0802 we have a MOM6_OUTPUT_FH defined, which is defined in forecast_predet.

I'm assuming those are consistent definitions?

[dpsarmie]

FHOUT_OCN_GFS / FHOUT_OCN is the variable that controls the MOM6 output frequency in GW and MOM6_OUTPUT_FH is just used to store an array of output times (in GW), correct? If so, then there might be an issue.

[JessicaMeixner-NOAA]

FHOUT_OCN_GFS / FHOUT_OCN is the variable that controls the MOM6 output frequency in GW and MOM6_OUTPUT_FH is just used to store an array of output times (in GW), correct? If so, then there might be an issue.

Yes. And looking a bit further, this variable should be "FHOUT_OCN" and not MOM6_OUTPUT_FH.

[ClaraDraper-NOAA]

I haven't tested it, but can confirm that this PR brings the needed update to CCPP/physics into UFS_UTILS to fix the gfs_sfcanl issue. It also updates the CCPP/physics used by the model to the same hash.

Note that updating the ufs_model and ufs_utils brings in many additional changes in addition to the gfs_sfcnl issue.

[DavidHuber-NOAA]

All tests passed on Ursa. I will run a develop C96C48mx500 case to verify the MOM6 output, then launch CI on all platforms.

[DavidHuber-NOAA]

I verified the 6-hour MOM6 forecast for the C96C48mx025_S2SW_gfs_cyc case was identical to last night's nightly run of the same case and also verified that MOM6 output was written every 6 hours as expected:

>cmp /scratch3/NCEPDEV/stmp/David.Huber/rt_4389/COMROOT/C96C48mx500_S2SW_cyc_gfs_4389/gfs.20211220/18/model/ocean/history/gfs.t18z.6hr_avg.f120.nc /scratch3/NCEPDEV/global/role.glopara/GFS_CI_CD/URSA/BUILDS/GITLAB/nightly_6ada5183_010726/RUNTESTS/COMROOT/C96C48mx500_S2SW_cyc_gfs_6ada5183-6840/gfs.20211220/18/model/ocean/history/gfs.t18z.6hr_avg.f120.nc
> echo $?
0

Proceeding with CI testing.

[DavidHuber-NOAA]

I am still wrong. The wavepostsbs metatask is already a dependency of the archiving jobs. Instead, we have a silent failure. This is in /scratch3/NCEPDEV/global/role.glopara/GFS_CI_CD/HERA/BUILDS/GITLAB/pr_cases_4389_f6105b9e_7163/RUNTESTS/COMROOT/C48mx500_3DVarAOWCDA_f6105b9e-7163/logs/2021032500/gfs_wavepostsbs_f015-f017.log during an MPMD section of the job at line 11403:

3: + wave_grib2_sbs.sh[88]/scratch3/NCEPDEV/global/role.glopara/GFS_CI_CD/HERA/BUILDS/GITLAB/pr_cases_4389_f6105b9e_7163/global-workflow/exec/gfs_ww3_grib.x
3: + wave_grib2_sbs.sh[89]export err=126
3: + wave_grib2_sbs.sh[89]err=126
3: + wave_grib2_sbs.sh[90][[ 126 -ne 0 ]]
3: + wave_grib2_sbs.sh[91]echo 'FATAL ERROR: gfs_ww3_grib.x returned non-zero status: 126; exiting!'
3: FATAL ERROR: gfs_ww3_grib.x returned non-zero status: 126; exiting!
3: + wave_grib2_sbs.sh[92]exit 126

Going to the end of the MPMD job at line 9891:

+ run_mpmd.sh[66]IFS=
+ run_mpmd.sh[66]read -r line
+ run_mpmd.sh[71]unset_strict
+ preamble.sh[45]set +eu
+ preamble.sh[46]set +o pipefail
+ run_mpmd.sh[73]srun -l --export=ALL --hint=nomultithread --multi-prog --output=mpmd.%j.%t.out -n 7 /scratch3/NCEPDEV/stmp/role.glopara/RUNDIRS/C48mx500_3DVarAOWCDA_f6105b9e-7163/gfs.2021032500/wavepostsbs_f017.82108/mpmd_cmdfile
+ run_mpmd.sh[74]err=0
+ run_mpmd.sh[75]set_strict
+ preamble.sh[35][[ YES == \Y\E\S ]]
+ preamble.sh[37]set -eu
+ preamble.sh[39]set -o pipefail
+ run_mpmd.sh[101][[ 0 -eq 0 ]]

It appears that run_mpmd.sh is not correctly catching failures of individual jobs.