Skip to content

[production/RRFS.v1] fix RRFS/REFS restart reproducibility and DEBUG crash issues for RRFSv1 operational implementation #2925

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
MatthewPyle-NOAA merged 24 commits into from
Jan 20, 2026
+682 −625
Merged

[production/RRFS.v1] fix RRFS/REFS restart reproducibility and DEBUG crash issues for RRFSv1 operational implementation #2925

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
24 commits
Select commit Hold shift + click to select a range
bb9e0dd
Update .gitmodules
DusanJovic-NOAA Apr 2, 2025
a84d6e2
Fix rrfs (conus13km) decomp and restart tests
DusanJovic-NOAA Apr 3, 2025
3bfa4fd
Merge pull request #15 from DusanJovic-NOAA/rrfs_v1_conus13km_tests
JiliDong-NOAA Jun 3, 2025
22514b1
update .gitmodules
Jun 3, 2025
e671d61
update FV3
Jun 3, 2025
baa6e5e
Update FV3
DusanJovic-NOAA Jun 12, 2025
3411cbe
Update FV3
DusanJovic-NOAA Jun 12, 2025
83d799b
update for rrfsens restart bitwise reproducibility
Aug 27, 2025
e9aa657
update FV3
Aug 28, 2025
70ec08b
Merge remote-tracking branch 'upstream/rrfs_v1_conus13km_tests' into …
Aug 28, 2025
3f2108f
revert FV3
Aug 28, 2025
0ffe28c
update FV3
Aug 28, 2025
04fd1a1
remove spaces
Aug 28, 2025
f8e8e7b
REFS ensemble restart fix (#43)
JiliDong-NOAA Aug 30, 2025
d61e720
Merge pull request #17 from DusanJovic-NOAA/rrfs_v1_conus13km_tests
JiliDong-NOAA Oct 7, 2025
900e5a7
update FV3 and stochastic_physics; fix DEBUG crash for REFS
Oct 7, 2025
2290f53
update FV3
Oct 8, 2025
b609aba
update FV3 to fix C3 compiling error
JiliDong-NOAA Oct 15, 2025
dbeb717
bl_date update
Oct 23, 2025
840b101
WCOSS2 RRFS Tests Pass
BrianCurtis-NOAA Oct 24, 2025
6b7e746
add hercules RT log: passed
jkbk2004 Nov 12, 2025
3787c33
remove unused conf file
Jan 20, 2026
79026eb
revert ufsatm and stochastic_physics back to community
Jan 20, 2026
3ddbc2e
update ufsatm and stochastic_physics
Jan 20, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion FV3
Open in desktop
Submodule FV3 updated from 81e6d1 to f861e4
2 changes: 1 addition & 1 deletion stochastic_physics
Open in desktop
Submodule stochastic_physics updated 4 files
+1 −1 get_stochy_pattern.F90
+17 −4 lndp_apply_perts.F90
+3 −3 stochy_data_mod.F90
+3 −3 stochy_patterngenerator.F90
2 changes: 1 addition & 1 deletion tests/bl_date.conf
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1 +1 @@
export BL_DATE=20250123
export BL_DATE=20251017
4 changes: 2 additions & 2 deletions tests/default_vars.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1402,8 +1402,8 @@ export SHAL_CNV=.false.
export DO_SAT_ADJ=.false.
export DO_DEEP=.false.
export CCPP_SUITE='FV3_HRRR'
export INPES=12
export JNPES=12
export INPES=10
export JNPES=8
export NPX=397
export NPY=233
export NPZ=65
Expand Down
7 changes: 4 additions & 3 deletions tests/fv3_conf/fv3_qsub.IN_acorn
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,15 +5,16 @@
#PBS -N @[JBNME]
#PBS -A @[ACCNR]
#PBS -q @[QUEUE]
#PBS -l place=vscatter,select=@[NODES]:ncpus=@[TPN]:mpiprocs=@[TPN]:mem=500G
#PBS -l place=vscatter,select=@[NODES]:ncpus=@[NCPUS]:mpiprocs=@[TPN]:ompthreads=@[THRD]:mem=500G
#PBS -l place=excl
#PBS -l walltime=00:@[WLCLK]:00

set -eux
echo -n " $( date +%s )," > job_timestamp.txt

cd $PBS_O_WORKDIR

echo -n " $( date +%s )," > job_timestamp.txt

set +x
module use $PWD/modulefiles
module load modules.fv3
Expand All @@ -30,7 +31,7 @@ export ESMF_RUNTIME_COMPLIANCECHECK=OFF:depth=4
export ESMF_RUNTIME_PROFILE=ON
export ESMF_RUNTIME_PROFILE_OUTPUT="SUMMARY"

mpiexec -n @[TASKS] -ppn @[TPN] -depth @[THRD] ./fv3.exe
mpiexec -n @[TASKS] -ppn @[TPN] --cpu-bind core -depth @[THRD] ./fv3.exe

echo "Model ended: " `date`
echo -n " $( date +%s )," >> job_timestamp.txt
7 changes: 4 additions & 3 deletions tests/fv3_conf/fv3_qsub.IN_wcoss2
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,15 +5,16 @@
#PBS -N @[JBNME]
#PBS -A @[ACCNR]
#PBS -q @[QUEUE]
#PBS -l place=vscatter,select=@[NODES]:ncpus=@[TPN]:mpiprocs=@[TPN]:mem=500G
#PBS -l place=vscatter,select=@[NODES]:ncpus=@[NCPUS]:mpiprocs=@[TPN]:ompthreads=@[THRD]:mem=500G
#PBS -l place=excl
#PBS -l walltime=00:@[WLCLK]:00

set -eux
echo -n " $( date +%s )," > job_timestamp.txt

cd $PBS_O_WORKDIR

echo -n " $( date +%s )," > job_timestamp.txt

set +x
module use $PWD/modulefiles
module load modules.fv3
Expand All @@ -30,7 +31,7 @@ export ESMF_RUNTIME_COMPLIANCECHECK=OFF:depth=4
export ESMF_RUNTIME_PROFILE=ON
export ESMF_RUNTIME_PROFILE_OUTPUT="SUMMARY"

mpiexec -n @[TASKS] -ppn @[TPN] -depth @[THRD] ./fv3.exe
mpiexec -n @[TASKS] -ppn @[TPN] --cpu-bind core -depth @[THRD] ./fv3.exe

echo "Model ended: " `date`
echo -n " $( date +%s )," >> job_timestamp.txt
604 changes: 317 additions & 287 deletions tests/logs/RegressionTests_hercules.log
View file
Open in desktop

Large diffs are not rendered by default.

600 changes: 315 additions & 285 deletions tests/logs/RegressionTests_wcoss2.log
View file
Open in desktop

Large diffs are not rendered by default.

29 changes: 9 additions & 20 deletions tests/rt.conf
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -199,12 +199,9 @@ RUN | hrrr_control_restart_dyn32_phy32 | - noaacloud

COMPILE | rrfs_dyn32_phy32_faster | intel | -DAPP=ATM -DFASTER=ON -DCCPP_SUITES=FV3_HRRR -D32BIT=ON -DCCPP_32BIT=ON | - noaacloud | fv3 |
RUN | conus13km_control | - noaacloud | baseline |
RUN | conus13km_2threads | - noaacloud | | conus13km_control
RUN | conus13km_restart_mismatch | - noaacloud | baseline | conus13km_control

# Expected to fail:
# RUN | conus13km_restart | - noaacloud | | conus13km_control
# RUN | conus13km_decomp | - noaacloud | | conus13km_control
RUN | conus13km_2threads | - noaacloud | |
RUN | conus13km_restart | - noaacloud | | conus13km_control
RUN | conus13km_decomp | - noaacloud | |

COMPILE | rrfs_dyn64_phy32 | intel | -DAPP=ATM -DCCPP_SUITES=FV3_RAP,FV3_HRRR -DCCPP_32BIT=ON | - noaacloud | fv3 |
RUN | rap_control_dyn64_phy32 | - noaacloud | baseline |
Expand All @@ -213,13 +210,10 @@ COMPILE | rrfs_dyn32_phy32_debug | intel | -DAPP=ATM -DCCPP_SUITES=FV3_RAP,FV3_H
RUN | rap_control_debug_dyn32_phy32 | - noaacloud | baseline |
RUN | hrrr_control_debug_dyn32_phy32 | - noaacloud | baseline |
RUN | conus13km_debug | - noaacloud | baseline |
RUN | conus13km_debug_qr | - derecho noaacloud | |
RUN | conus13km_debug_2threads | - derecho noaacloud | |
RUN | conus13km_debug_qr | - noaacloud | |
RUN | conus13km_debug_2threads | - noaacloud | |
RUN | conus13km_radar_tten_debug | - noaacloud | baseline |

# Expected to fail:
# RUN | conus13km_debug_decomp | - noaacloud | |

COMPILE | rrfs_dyn64_phy32_debug | intel | -DAPP=ATM -DCCPP_SUITES=FV3_RAP,FV3_HRRR -DCCPP_32BIT=ON -DDEBUG=ON | - noaacloud | fv3 |
RUN | rap_control_dyn64_phy32_debug | - noaacloud | baseline |

Expand Down Expand Up @@ -371,12 +365,9 @@ RUN | hrrr_control_decomp_dyn32_phy32 | + hera hercules
RUN | rap_restart_dyn32_phy32 | + hera hercules | | rap_control_dyn32_phy32
RUN | hrrr_control_restart_dyn32_phy32 | + hera hercules | | hrrr_control_dyn32_phy32
RUN | conus13km_control | + hera hercules | baseline |
RUN | conus13km_2threads | + hera hercules | | conus13km_control
RUN | conus13km_restart_mismatch | + hera hercules | baseline | conus13km_control

# Expected to fail:
# RUN | conus13km_restart | + hera hercules | | conus13km_control
# RUN | conus13km_decomp | + hera hercules | | conus13km_control
RUN | conus13km_2threads | + hera hercules | |
RUN | conus13km_decomp | + hera hercules | |
RUN | conus13km_restart | + hera hercules | | conus13km_control

COMPILE | atm_dyn64_phy32 | gnu | -DAPP=ATM -DCCPP_32BIT=ON | + hera hercules | fv3 |
RUN | rap_control_dyn64_phy32 | + hera hercules | baseline |
Expand All @@ -387,11 +378,9 @@ RUN | hrrr_control_debug_dyn32_phy32 | + hera hercules
RUN | conus13km_debug | + hera hercules | baseline |
RUN | conus13km_debug_qr | + hera hercules | |
RUN | conus13km_debug_2threads | + hera hercules | |
RUN | conus13km_debug_decomp | + hera hercules | |
RUN | conus13km_radar_tten_debug | + hera hercules | baseline |

# Expected to fail:
# RUN | conus13km_debug_decomp | + hera derecho hercules | |

COMPILE | atm_dyn64_phy32_debug | gnu | -DAPP=ATM -DCCPP_32BIT=ON -DDEBUG=ON | + hera hercules | fv3 |
RUN | rap_control_dyn64_phy32_debug | + hera hercules | baseline |

Expand Down
13 changes: 4 additions & 9 deletions tests/rt.conf_rrfs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -86,12 +86,9 @@ RUN | hrrr_control_restart_dyn32_phy32 | - noaacloud

COMPILE | rrfs_dyn32_phy32_faster | intel | -DAPP=ATM -DFASTER=ON -DCCPP_SUITES=FV3_HRRR -D32BIT=ON -DCCPP_32BIT=ON -DENABLE_PARALLELRESTART=NO -DENABLE_RRFS_WAR=NO | - noaacloud | fv3 |
RUN | conus13km_control | - noaacloud | baseline |
RUN | conus13km_2threads | - noaacloud | | conus13km_control
RUN | conus13km_restart_mismatch | - noaacloud | baseline | conus13km_control

# Expected to fail:
# RUN | conus13km_restart | - noaacloud | | conus13km_control
# RUN | conus13km_decomp | - noaacloud | | conus13km_control
RUN | conus13km_2threads | - noaacloud | |
RUN | conus13km_decomp | - noaacloud | |
RUN | conus13km_restart | - noaacloud | | conus13km_control

COMPILE | rrfs_dyn64_phy32 | intel | -DAPP=ATM -DCCPP_SUITES=FV3_RAP,FV3_HRRR -DCCPP_32BIT=ON -DENABLE_PARALLELRESTART=NO -DENABLE_RRFS_WAR=NO | - noaacloud | fv3 |
RUN | rap_control_dyn64_phy32 | - noaacloud | baseline |
Expand All @@ -102,11 +99,9 @@ RUN | hrrr_control_debug_dyn32_phy32 | - noaacloud
RUN | conus13km_debug | - noaacloud | baseline |
RUN | conus13km_debug_qr | - derecho noaacloud | |
RUN | conus13km_debug_2threads | - derecho noaacloud | |
RUN | conus13km_debug_decomp | - noaacloud | |
RUN | conus13km_radar_tten_debug | - noaacloud | baseline |

# Expected to fail:
# RUN | conus13km_debug_decomp | - noaacloud | |

COMPILE | rrfs_dyn64_phy32_debug | intel | -DAPP=ATM -DCCPP_SUITES=FV3_RAP,FV3_HRRR -DCCPP_32BIT=ON -DDEBUG=ON -DENABLE_PARALLELRESTART=NO -DENABLE_RRFS_WAR=NO | - noaacloud | fv3 |
RUN | rap_control_dyn64_phy32_debug | - noaacloud | baseline |

Expand Down
2 changes: 2 additions & 0 deletions tests/run_test.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -312,6 +312,8 @@ fi
export PPN
export UFS_TASKS

export NCPUS=$(( TPN * THRD ))

if [[ $SCHEDULER = 'pbs' ]]; then
if [[ -e $PATHRT/fv3_conf/fv3_qsub.IN_${MACHINE_ID} ]]; then
atparse < $PATHRT/fv3_conf/fv3_qsub.IN_${MACHINE_ID} > job_card
Expand Down
9 changes: 5 additions & 4 deletions tests/tests/conus13km_2threads
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -17,11 +17,12 @@ export LIST_FILES="sfcf000.nc \
atmf001.nc"

export_hrrr_conus13km
export SMONTH=5
export RESTART_INTERVAL=1
export FHMAX=1
export atm_omp_num_threads=2
export INPES=12
export JNPES=12
export WRTTASK_PER_GROUP=6
export QUILTING_RESTART=.false.
export USE_MERRA2=.true.

if [[ " hera orion hercules jet " =~ " ${MACHINE_ID} " ]] ; then
ZSTANDARD_LEVEL=5
fi
3 changes: 0 additions & 3 deletions tests/tests/conus13km_debug_2threads
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -20,7 +20,4 @@ export_hrrr_conus13km
export RESTART_INTERVAL=1
export FHMAX=1
export atm_omp_num_threads=2
export INPES=12
export JNPES=12
export WRTTASK_PER_GROUP=6
export QUILTING_RESTART=.false.
4 changes: 2 additions & 2 deletions tests/tests/conus13km_debug_decomp
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -19,6 +19,6 @@ export LIST_FILES="sfcf000.nc \
export_hrrr_conus13km
export RESTART_INTERVAL=1
export FHMAX=1
export INPES=16
export JNPES=9
export INPES=8
export JNPES=10
export QUILTING_RESTART=.false.
5 changes: 3 additions & 2 deletions tests/tests/conus13km_debug_qr
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,13 +1,13 @@
###############################################################################
#
# HRRR physics on 13km domain, control
# HRRR physics on 13km domain, debug with quilting restart
#
###############################################################################

# This configuration is supposed to match the rrfs_dev1 parallel, so
# do not change the namelist without checking the current parallel.

export TEST_DESCR="HRRR physics on 13km domain, control"
export TEST_DESCR="HRRR physics on 13km domain, debug with quilting restart"

export CNTL_DIR=conus13km_debug

Expand All @@ -26,3 +26,4 @@ export LIST_FILES="sfcf000.nc \
export_hrrr_conus13km
export RESTART_INTERVAL=1
export FHMAX=1
export QUILTING_RESTART=.true.
9 changes: 7 additions & 2 deletions tests/tests/conus13km_decomp
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -20,6 +20,11 @@ export LIST_FILES="sfcf000.nc \

export_hrrr_conus13km
export RESTART_INTERVAL=1
export INPES=16
export JNPES=9
export INPES=8
export JNPES=10
export QUILTING_RESTART=.false.
export USE_MERRA2=.true.

if [[ " hera orion hercules jet " =~ " ${MACHINE_ID} " ]] ; then
ZSTANDARD_LEVEL=5
fi
5 changes: 5 additions & 0 deletions tests/tests/conus13km_restart
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -19,3 +19,8 @@ export FHROT=1
export RESTART_FILE_PREFIX="${SYEAR}${SMONTH}${SDAY}.$(printf "%02d" $(( ${SHOUR} + ${FHROT} )))0000"
export RRFS_RESTART=YES
export QUILTING_RESTART=.false.
export USE_MERRA2=.true.

if [[ " hera orion hercules jet " =~ " ${MACHINE_ID} " ]] ; then
ZSTANDARD_LEVEL=5
fi