[production/RRFS.v1] fix RRFS/REFS prod v1 restart reproducibility and DEBUG mode crash for operation implementation

[JiliDong-NOAA]

Commit Queue Requirements:

• This PR addresses a relevant WM issue (if not, create an issue).
• All subcomponent pull requests (if any) have been reviewed by their code managers.
• Run the full Intel+GNU RT suite (compared to current baselines), preferably on Ursa (Derecho or Hercules are acceptable alternatives). Exceptions: documentation-only PRs, CI-only PRs, etc.
  • Commit log file w/full results from RT suite run (if applicable).
  • Commit 'test_changes.list' (unless file is empty).
• Fill out all sections of this template.

---

Description:

This PR addresses:

1. RRFS/REFS restart reproducibility when cumulus convection is applied, including saSAS and Grell-Freitas
2. REFS ensemble restart reproducibility issues when running with 32 bit physics by
   changing the SPP random seeds and spectral coefficient variable names for consistency when reading these variables for restart (spppert_seed->spp_seed; ppcpert_spec->spp_spec)
   modifying data type from single to double precision when reading and writing the above two variables
3. crash when running REFS under DEBUG mode

Commit Message:

* UFSWM - 
  * AQM - 
  * CDEPS - 
  * CICE - 
  * CMEPS - 
  * CMakeModules - 
  * UFSATM - 
    * ccpp-physics - 
    * atmos_cubed_sphere - 
  * GOCART - 
  * HYCOM - 
  * MOM6 - 
  * NOAHMP - 
  * WW3 - 
  * fire_behavior
  * stochastic_physics - 

Priority:

• Critical Bugfix: Reason
• High: Reason
• Normal

Git Tracking

UFSWM:

• Closes #

Sub component Pull Requests:

• AQM:
• CDEPS:
• CICE:
• CMEPS:
• CMakeModules:
• FV3: [production/RRFS.v1] update ccpp/physics for RRFSv1 implementation NOAA-EMC/ufsatm#1016
  • ccpp-physics: [production/RRFS.v1] RRFS restart reproducibility fix ccpp-physics#315
  • atmos_cubed_sphere:
• GOCART:
• HYCOM:
• MOM6:
• NOAHMP:
• WW3:
• fire_behavior:
• stochastic_physics: [production/RRFS.v1]Fixes for REFS restart reproducibility and crash under DEBUG NOAA-PSL/stochastic_physics#90
• None

UFSWM Blocking Dependencies:

• Blocked by #
• None

Documentation:

• Documentation update required.
  • Relevant updates are included with this PR.
  • A WM issue has been opened to track the need for a documentation update; a person responsible for submitting the update has been assigned to the issue (link issue).
• Documentation update NOT required.
  • Explanation:

---

Changes

Regression Test Changes (Please commit test_changes.list):

• PR Adds New Tests/Baselines.
• PR Updates/Changes Baselines.
• No Baseline Changes.

Input data Changes:

• None.
• New input data.
• Updated input data.

Library Changes/Upgrades:

• Required
  • Library names w/versions:
  • Git Stack Issue (JCSDA/spack-stack#)
• No Updates

---

Testing Log:

• RDHPCS
  • Hera
  • Orion
  • Hercules
  • GaeaC6
  • Derecho
  • Ursa
• WCOSS2
  • Dogwood/Cactus
  • Acorn
• CI
• opnReqTest (complete task if unnecessary)

[gspetro-NOAA]

@JiliDong-NOAA Is this PR ready? If so, we need you to fill out the PR template to convey the information that CMs need to schedule and process it.

[DeniseWorthen]

@gspetro-NOAA This is a PR to a production branch, so I think a different criteria apply in terms of testing etc.

[gspetro-NOAA]

@DeniseWorthen Agreed, but it's still not clear what testing has been done/whether it's ready to go because there's nothing in the PR description that speaks to that. Plenty of people open a PR that they're still doing a little work on, so I thought it was best just to check in/confirm. Will add it to the list to bring up at tomorrow's CM meeting, but it would still be good to have a confirmation from the developers that no further work is required and that they have finished testing it on their side.

[JiliDong-NOAA]

Most of the work has been completed for RRFS implementation but this PR only includes changes to part of submodules (ccpp-physics and stochastic physics). I am considering putting this PR on hold and combine everything together for one single ufs-weather-model PR. I will coordinate with EMC colleagues and do that tomorrow.

[gspetro-NOAA]

@JiliDong-NOAA Thanks for the update! Just let us know if/when you're ready to proceed!

[DeniseWorthen]

@gspetro-NOAA As far as I'm aware, no RTs are actually run when PRs go to a production branch, which was the gist of my comment. When the updates are finally brought into develop, that is when they are tested etc. Still from the comment above, it may be a moot point.

[JiliDong-NOAA]

Thanks @gspetro-NOAA and @DeniseWorthen for your comments. I am closing this one with new PR opened #2925
Please let me know if there is anything I need to do.