Skip to content

Fix 06z gfs_sfcanl jobs failing due to a bug in CCPP #4389

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
DavidHuber-NOAA merged 18 commits into from
Jan 20, 2026
Merged

Fix 06z gfs_sfcanl jobs failing due to a bug in CCPP #4389

DavidHuber-NOAA merged 18 commits into from
Jan 20, 2026
+20 −2

Conversation

@DavidHuber-NOAA
Copy link
Contributor

@DavidHuber-NOAA DavidHuber-NOAA commented Dec 29, 2025 •
edited
Loading

Description

This fixes a bug in CCPP that prevented the 06Z gfs_sfcanl job from running global_cycle on the 03Z IAU time. The code change now allows the input hour to be any integer between 0 and 23.
Resolves #4364
Refs #4408 (partially resolves but a full investigation is needed)

Type of change

  • Bug fix (fixes something broken)
  • New feature (adds functionality)
  • Maintenance (code refactor, clean-up, new CI test, etc.)

Change characteristics

How has this been tested?

  • C96_atm3DVar_extended test on WCOSS2
  • UFS_Utils regression tests on Ursa (no change to baseline)
  • UFS regression tests
  • Full suite of GW tests on all platforms (when the UFS model hash is ready)

Checklist

  • Any dependent changes have been merged and published
  • My code follows the style guidelines of this project
  • I have performed a self-review of my own code
  • I have commented my code, particularly in hard-to-understand areas
  • I have documented my code, including function, input, and output descriptions
  • My changes generate no new warnings
  • New and existing tests pass with my changes
  • This change is covered by an existing CI test or a new one has been added

@DavidHuber-NOAA DavidHuber-NOAA added the GFS Change This PR, if merged, will change results for the GFS. label Dec 29, 2025
@DavidHuber-NOAA DavidHuber-NOAA changed the title Fix/gfs sfcanl Fix 06z gfs_sfcanl jobs failing due to a bug in CCPP Dec 29, 2025
@RussTreadon-NOAA RussTreadon-NOAA mentioned this pull request Jan 5, 2026
Merged
26 tasks
@DavidHuber-NOAA DavidHuber-NOAA marked this pull request as ready for review January 6, 2026 20:30
aerorahul
aerorahul reviewed Jan 6, 2026
View reviewed changes
ush/parsing_ufs_configure.sh Outdated
Comment on lines 72 to 73
local MOM6_OUTPUT_DIR="${MOM6_OUTPUT_DIR:-./MOM6_OUTPUT}"
local MOM6_RESTART_DIR="${MOM6_RESTART_DIR:-./MOM6_RESTART}"
Copy link
Contributor

@aerorahul aerorahul Jan 6, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
local MOM6_OUTPUT_DIR="${MOM6_OUTPUT_DIR:-./MOM6_OUTPUT}"
local MOM6_RESTART_DIR="${MOM6_RESTART_DIR:-./MOM6_RESTART}"
local MOM6_OUTPUT_DIR="./MOM6_OUTPUT"
local MOM6_RESTART_DIR="./MOM6_RESTART"

This isn't an option. The workflow will always write to this space. The other instances where this variable is set in this manner is ush/parsing_namelists_FV3.sh

Copy link
Contributor

@JessicaMeixner-NOAA JessicaMeixner-NOAA Jan 6, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I agree, I don't think this should be configurable. This is assumed other places.

I also see that ufs-community/ufs-weather-model@6f22f57...f8b0802 we have a MOM6_OUTPUT_FH defined, which is defined in forecast_predet.

I'm assuming those are consistent definitions?

aerorahul
aerorahul previously approved these changes Jan 6, 2026
View reviewed changes
Copy link
Contributor

@aerorahul aerorahul left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

just one suggestion, looks good.

JessicaMeixner-NOAA
JessicaMeixner-NOAA reviewed Jan 6, 2026
View reviewed changes
Copy link
Contributor

@JessicaMeixner-NOAA JessicaMeixner-NOAA left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We should run a S2S test of some sort to make sure ocean output looks okay given the changes. I didn't follow exactly when those changes went in. @jiandewang or @dpsarmie might know more about that the MOM6_OUTPUT_FH to make sure that is as expected here now.

ush/parsing_ufs_configure.sh Outdated
Comment on lines 72 to 73
local MOM6_OUTPUT_DIR="${MOM6_OUTPUT_DIR:-./MOM6_OUTPUT}"
local MOM6_RESTART_DIR="${MOM6_RESTART_DIR:-./MOM6_RESTART}"
Copy link
Contributor

@JessicaMeixner-NOAA JessicaMeixner-NOAA Jan 6, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I agree, I don't think this should be configurable. This is assumed other places.

I also see that ufs-community/ufs-weather-model@6f22f57...f8b0802 we have a MOM6_OUTPUT_FH defined, which is defined in forecast_predet.

I'm assuming those are consistent definitions?

@dpsarmie
Copy link
Contributor

dpsarmie commented Jan 6, 2026

FHOUT_OCN_GFS / FHOUT_OCN is the variable that controls the MOM6 output frequency in GW and MOM6_OUTPUT_FH is just used to store an array of output times (in GW), correct? If so, then there might be an issue.

@JessicaMeixner-NOAA
Copy link
Contributor

JessicaMeixner-NOAA commented Jan 6, 2026

FHOUT_OCN_GFS / FHOUT_OCN is the variable that controls the MOM6 output frequency in GW and MOM6_OUTPUT_FH is just used to store an array of output times (in GW), correct? If so, then there might be an issue.

Yes. And looking a bit further, this variable should be "FHOUT_OCN" and not MOM6_OUTPUT_FH.

ClaraDraper-NOAA
ClaraDraper-NOAA previously approved these changes Jan 6, 2026
View reviewed changes
Copy link
Contributor

@ClaraDraper-NOAA ClaraDraper-NOAA left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I haven't tested it, but can confirm that this PR brings the needed update to CCPP/physics into UFS_UTILS to fix the gfs_sfcanl issue. It also updates the CCPP/physics used by the model to the same hash.

Note that updating the ufs_model and ufs_utils brings in many additional changes in addition to the gfs_sfcnl issue.

@DavidHuber-NOAA
Copy link
Contributor Author

DavidHuber-NOAA commented Jan 8, 2026

All tests passed on Ursa. I will run a develop C96C48mx500 case to verify the MOM6 output, then launch CI on all platforms.

@ClaraDraper-NOAA ClaraDraper-NOAA mentioned this pull request Jan 8, 2026
Open
@DavidHuber-NOAA
Copy link
Contributor Author

DavidHuber-NOAA commented Jan 8, 2026

I verified the 6-hour MOM6 forecast for the C96C48mx025_S2SW_gfs_cyc case was identical to last night's nightly run of the same case and also verified that MOM6 output was written every 6 hours as expected:

>cmp /scratch3/NCEPDEV/stmp/David.Huber/rt_4389/COMROOT/C96C48mx500_S2SW_cyc_gfs_4389/gfs.20211220/18/model/ocean/history/gfs.t18z.6hr_avg.f120.nc /scratch3/NCEPDEV/global/role.glopara/GFS_CI_CD/URSA/BUILDS/GITLAB/nightly_6ada5183_010726/RUNTESTS/COMROOT/C96C48mx500_S2SW_cyc_gfs_6ada5183-6840/gfs.20211220/18/model/ocean/history/gfs.t18z.6hr_avg.f120.nc
> echo $?
0

Proceeding with CI testing.

@emcbot emcbot added CI-Ursa-Building **Bot use only** CI testing is cloning/building on Ursa CI-Hera-Building **Bot use only** CI testing is cloning/building on Hera CI-Gaeac6-Building **Bot use only** CI testing is cloning/building on Gaea C6 CI-Ursa-Running **Bot use only** CI testing on Ursa for this PR is in-progress CI-Hera-Running **Bot use only** CI testing on Hera for this PR is in-progress CI-Gaeac6-Running **Bot use only** CI testing on Gaea C6 for this PR is in-progress CI-Ursa-Passed **Bot use only** CI testing on Ursa for this PR has completed successfully CI-Hera-Passed **Bot use only** CI testing on Hera for this PR has completed successfully CI-Gaeac6-Passed **Bot use only** CI testing on Gaea C6 for this PR has completed successfully and removed CI-Ursa-Ready **CM use only** PR is ready for CI testing on Ursa CI-Hera-Ready **CM use only** PR is ready for CI testing on Hera CI-Gaeac6-Ready **CM use only** PR is ready for CI testing on Gaea C6 CI-Ursa-Building **Bot use only** CI testing is cloning/building on Ursa CI-Hera-Building **Bot use only** CI testing is cloning/building on Hera CI-Gaeac6-Building **Bot use only** CI testing is cloning/building on Gaea C6 CI-Ursa-Running **Bot use only** CI testing on Ursa for this PR is in-progress CI-Hera-Running **Bot use only** CI testing on Hera for this PR is in-progress CI-Gaeac6-Running **Bot use only** CI testing on Gaea C6 for this PR is in-progress labels Jan 16, 2026
@DavidHuber-NOAA DavidHuber-NOAA added CI-Wcoss2-Passed (CM) Manual CI testing on WCOSS for this PR has completed successfully CI-Wcoss2-Passed CI testing on WCOSS for this PR has completed successfully and removed CI-Wcoss2-Running CI testing on WCOSS for this PR is in-progress CI-Wcoss2-Passed (CM) Manual CI testing on WCOSS for this PR has completed successfully labels Jan 20, 2026
@DavidHuber-NOAA
Copy link
Contributor Author

DavidHuber-NOAA commented Jan 20, 2026

All tests passed on all platforms. Requesting final approvals to merge.

JessicaMeixner-NOAA
JessicaMeixner-NOAA approved these changes Jan 20, 2026
View reviewed changes
dev/workflow/rocoto/gfs_tasks.py
dep_dict = {'type': 'task', 'name': f'{self.run}_fbwind'}
deps.append(rocoto.add_dependency(dep_dict))

if self.options['do_wave']:
Copy link
Contributor

@JessicaMeixner-NOAA JessicaMeixner-NOAA Jan 20, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is okay to add but we don't actually archive any of the awips files for waves to my knowledge, so I don't think this is actually needed.

Copy link
Contributor Author

@DavidHuber-NOAA DavidHuber-NOAA Jan 20, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good to know! I'll open an issue to remove this tarball/job.

Copy link
Contributor Author

@DavidHuber-NOAA DavidHuber-NOAA Jan 20, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Opened issues #4454 to retire AWIPS waves archiving.

@DavidHuber-NOAA DavidHuber-NOAA mentioned this pull request Jan 20, 2026
Closed
4 tasks
@DavidHuber-NOAA DavidHuber-NOAA merged commit 53dc401 into NOAA-EMC:develop Jan 20, 2026
5 checks passed
weihuang-jedi added a commit to NOAA-EPIC/global-workflow-cloud that referenced this pull request Jan 22, 2026
@weihuang-jedi
Merge branch 'feature/use_container_prefix' of github.com:NOAA-EPIC/g…
936e7bd 
…lobal-workflow-cloud into feature/use_container_prefix

* 'feature/use_container_prefix' of github.com:NOAA-EPIC/global-workflow-cloud:
  sync with EMC build_all.sh fix
  Fix 06z gfs_sfcanl jobs failing due to a bug in CCPP (NOAA-EMC#4389)
  Build on compute or login nodes from one build script. (NOAA-EMC#4380)
  Refactor build_compute.py to optionally support node exclusion (NOAA-EMC#4447)
  Fix incorrect branching in bash code anl (NOAA-EMC#4442)
  Update of GitHub Copilot Instructions file for current MCP/RAG system (NOAA-EMC#4397)
  Fix reviewdog arg name (NOAA-EMC#4441)
  Fix bash code analysis issues (NOAA-EMC#4440)
  Enable gcafs tests on Ursa (NOAA-EMC#4439)
  Initialize LETKF solver after running observer (NOAA-EMC#4424)
  Move C96mx025 case to sfsv1 (NOAA-EMC#4435)
  Update HPC support levels. (NOAA-EMC#4430)
  Adding GCAFS products v2  (NOAA-EMC#4407)
  Feature/task template (NOAA-EMC#4427)
  Run GEMPAK jobs on prepost nodes (NOAA-EMC#4411)
  Build the GDASApp with 8 cores on Gaea again
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@JessicaMeixner-NOAA JessicaMeixner-NOAA JessicaMeixner-NOAA approved these changes

@ClaraDraper-NOAA ClaraDraper-NOAA ClaraDraper-NOAA left review comments

@RuiyuSun RuiyuSun Awaiting requested review from RuiyuSun

@CatherineThomas-NOAA CatherineThomas-NOAA Awaiting requested review from CatherineThomas-NOAA

@junwang-noaa junwang-noaa Awaiting requested review from junwang-noaa junwang-noaa is a code owner

@jiandewang jiandewang Awaiting requested review from jiandewang

@aerorahul aerorahul Awaiting requested review from aerorahul aerorahul is a code owner

Assignees

No one assigned

Labels

CI-Gaeac6-Passed **Bot use only** CI testing on Gaea C6 for this PR has completed successfully CI-Hera-Passed **Bot use only** CI testing on Hera for this PR has completed successfully CI-Hercules-Passed **Bot use only** CI testing on Hercules for this PR has completed successfully CI-Orion-Passed **Bot use only** CI testing on Orion for this PR has completed successfully CI-Ursa-Passed **Bot use only** CI testing on Ursa for this PR has completed successfully CI-Wcoss2-Passed CI testing on WCOSS for this PR has completed successfully GFS Change This PR, if merged, will change results for the GFS.

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

global_cycle reports a fatal error on 06Z gfs cycles

7 participants

@DavidHuber-NOAA @dpsarmie @JessicaMeixner-NOAA @DeniseWorthen @emcbot @aerorahul @ClaraDraper-NOAA