Set up model run on WCOSS

@DanIredell-NOAA

Goal:

Set up the model run for RTOFS v2.3 parallel testing on the WCOSS platform.

Tasks:

Prepare necessary input datasets and initial conditions.
Validate runtime and diagnose timing performance from:
- Entire model.
- And its parts, namely: DATM, MOM6 and CICE6.
- Keep track of IO.

Related parent issue: Set up a twin like RTOFS v2.5

Starting point:

@DanIredell-NOAA:

Work directory: /lfs/h2/emc/ptmp/dan.iredell/rb.v3.0/NC-v2.5.2/20250503/rtofs_mom.1499882
Out and err /lfs/h2/emc/couple/noscrub/dan.iredell/COMDIR/logs/20250503/rtofs_mom.o225806654
Job card /lfs/h2/emc/ptmp/dan.iredell/rb.v3.0/batchscripts/rtofs.mom.1499882
Notes:

ranks 0-79 are for cmeps and datm. rank 80-1319 for ocn (mom6).  Rank 80 is where it crashed in esmf (I assume ocn-init)

The ⬆️ originate from: /scratch3/NCEPDEV/marine/Zulema.Garraffo/FV3_RT/expt_14.4_hera/rtofs.20250503 on Hera/Ursa; out and err are in that directory.

Next steps:

@DeniseWorthen:

Notes:

So it's failing at the line in the mom_cap where it reads in the mesh; but the same mesh is used for cice, and I can see that it successfully reads that in. So it doesn't seem like it would be a problem w/ the mesh file.

We have new meshes for the DATM configuration which no longer contain land. 

You need to change datm_in:
  model_maskfile = "INPUT/[gfs_mesh.nc](http://gfs_mesh.nc/)"
  model_meshfile = "INPUT/[gfs_mesh.nc](http://gfs_mesh.nc/)"

ufs.configure
mesh_atm  = INPUT/[gfs_mesh.nc](http://gfs_mesh.nc/)

instead of [gfs_mesh.nc](http://gfs_mesh.nc/), you'll need the file name [mesh.datm.3072x1536.nc](http://mesh.datm.3072x1536.nc/) 

You can copy it from:

/scratch3/NAGAPE/epic/role.epic/UFS-WM_RT/NEMSfv3gfs/input-data-20250507/DATM_CDEPS/[mesh.datm.3072x1536.nc](http://mesh.datm.3072x1536.nc/)

@DanIredell-NOAA changed "mesh file"

workdir: /lfs/h2/emc/ptmp/dan.iredell/rb.v3.0/NC-v2.5.2/20250503/rtofs_mom.967099/
out and err: /lfs/h2/emc/couple/noscrub/dan.iredell/COMDIR/logs/20250503/rtofs_mom.o225898219

@DeniseWorthen:
I have something that ran for 6 hours: /lfs/h2/emc/nems/noscrub/denise.worthen/FV3_RT/rt_4045468/datm.mx008.rundir

I had to really up the resources though and right now cice history is turned off. One thing is that there are

(zap_snow_temperature)zap_snow_temperature: temperature out of bounds!
messages in the stdout.

Suggested fix to eliminate: you have both sea ice on land, and a bunch of corrupted sea ice values:

The other thing to check for is whether there is snow or ice on land in the restart file

if (sum(aicen) > 0 .and. kmt == 0) then sea ice exists on land; same for vicen and vsnon

Necessary check(s):

There are a bunch of 9.969E+36 aicen values on land along the left hand-size of the domain i<500 and a bunch of sea ice on land in the Barents Sea

Which has been made from: this is from the [iced.2025-05-01-64800.nc](http://iced.2025-05-01-64800.nc/) which is being used as the IC file for CICE, as far as I can tell from how the it was set up in Dan's run directory.

To do:

In MOM_input, the big thing I had to change was the stagger. The stagger values (from coupler) need to be on A grid, not C. There are two of these; can't remember their names in MOM_input right now.
Route handle for run off:

Another thing you may want to track as an issue---creating the routehandles takes a LONG time. There are two ways to solve this. If you know your tasking (ie, PE counts) are fixed, then you can generate them the first time and then re-use them. But that only works when you're not changing your task counts. 

A second way is use mapfiles in CMEPS; we don't currently have that capability, but I was going to add it for GFSv17 but ran into a snag (details are unimportant here). The advantage of mapfiles is that you can re-use them even if you task counts change. I can pick that work up and get it working for the mx008 case when using a DATM.

Will wait for option 2 because @DeniseWorthen has most of the work done or almost there ..

I've looked at the grid_cice_NEMS_mx008.nc file sourced from "somewhere". The angle fields are wrong--they suffer from a long-standing bug which was fixed in ufs-community/UFS_UTILS#921.

I will be updating UFS-UTILS to create the correct (current) mesh files and grid files for CICE at 1/12 deg (ufs-community/UFS_UTILS#1046).

@DeniseWorthen

I've looked at the grid_cice_NEMS_mx008.nc file sourced from "somewhere". The angle fields are wrong--they suffer from a long-standing bug which was fixed in ufs-community/UFS_UTILS#921.

I will be updating UFS-UTILS to create the correct (current) mesh files and grid files for CICE at 1/12 deg (ufs-community/UFS_UTILS#1046).

Thank you @DeniseWorthen

You have everything needed to generate it?

I believe so. I only need the supergrid, the mask and the depths. The files I have are identical to what was in Dan's run directory.

I believe so. I only need the supergrid, the mask and the depths. The files I have are identical to what was in Dan's run directory.

Perfect!

@DeniseWorthen

@DeniseWorthen
2 things came to my mind while raking 🍃 !
(Not sure if they impact anything you planning on doing.)

(I assume, this point is implicit (yet mentioning anyway!).) The resolution of the atmospheric forcing, we'll use the corresponds to GFS (Gaussian grid) files

dimensions:
        latitude = 1536 ;
        longitude = 3072 ;

That's exactly what you have:

santha.akella@clogin06:~> grep '_global' /lfs/h2/emc/nems/noscrub/denise.worthen/FV3_RT/rt_4045468/datm.mx008.rundir/datm_in
  nx_global = 3072
  ny_global = 1536

👍

Question: You pointed me to this option/script for mesh file generation
Will the script or anything change?

2.a Is there a distinction between GFS and GEFS in the datm? Specifically in the following settings of datamode and case_name:

santha.akella@clogin06:/lfs/h2/emc/nems/noscrub/denise.worthen/FV3_RT/rt_4045468/datm.mx008.rundir> grep 'mode' datm_in
  datamode = "GEFS"

santha.akella@clogin06:/lfs/h2/emc/nems/noscrub/denise.worthen/FV3_RT/rt_4045468/datm.mx008.rundir> grep 'gfs' ufs.configure
      case_name = datm.gfs

In other words, should datamode be set to "GFS"? Checked if that's an option, found datm/datm_datamode_gefs_mod.F90 but none for gfs: datm/datm_datamode_gfs_mod.F90!

2.b Are we treat gefs and gfs as interchangeable key words in certain context(s) like the ⬆️?

@sanAkel

@sanAkel The "datamode" needs to be GEFS for any of our DATM configs (previously there had been multiple modes, but this was all consolidated a while ago). This is the CDEPS setting which toggles between possible data atmosphere sources:

    ! Validate sdat datamode
    if (mainproc) write(logunit,*) ' datm datamode = ',trim(datamode)
    if ( trim(datamode) == 'CORE2_NYF'    .or. &
         trim(datamode) == 'CORE2_IAF'    .or. &
         trim(datamode) == 'CORE_IAF_JRA' .or. &
         trim(datamode) == 'CLMNCEP'      .or. &
         trim(datamode) == 'CPLHIST'      .or. &
         trim(datamode) == 'GEFS'         .or. &
         trim(datamode) == 'ERA5'         .or. &
         trim(datamode) == 'SIMPLE') then

The case_name in ufs.configure just sets the file prefix for the restart (or mediator history) files, eg DATM_GEFS_NEW.datm.r.2011-10-02-00000.nc. I don't like the current defaults (all caps) so out of habit I change to to lower case.

Please see this directory for updated grid files: /scratch4/NCEPDEV/stmp/Denise.Worthen/CPLD_GRIDGEN/rt_1057019/008

grid_cice_NEMS_mx008.nc
kmtu_cice_NEMS_mx008.nc
mesh.mx008.nc

The grid_cice file contains the angle fix; the others should be updated for consistency.

@DeniseWorthen

Thanks @DeniseWorthen

I copied them to /scratch4/NCEPDEV/marine/Santha.Akella/to_test.
Will test on wcoss-2 tomorrow.
If there are issues, will alert you.

From the CICE documentation:

The grid is constructed by “folding” the top row, so that the left-hand half and the right-hand half of it coincide. Two choices for constructing the tripole grid are available. The one first introduced to CICE is called “U-fold”, which means that the poles and the grid cells between them are U cells on the grid. Alternatively the poles and the cells between them can be grid T cells, making a “T-fold.”

The files created by the UFS-UTILS cpld_gridgen utility, which reads the MOM6 ocean_hgrid.nc and writes out the CICE grid files and the mesh file show max nodecoord of +90.

The ocean domain that is created at runtime (looking at one of the ocean history files) shows a max geolat_c (corner) of +90.

Everything is consistent with the ice_in namelist setting tripole. The domain is folded along the U-row (CICE) or in MOM6 terms, the "Bu" points. The actual pole lies at a corner grid point.

If the pole was located on T-cells, you'd have a T-fold and the namelist setting would need to be tripoleT.

@DeniseWorthen

From the CICE documentation:

The grid is constructed by “folding” the top row, so that the left-hand half and the right-hand half of it coincide. Two choices for constructing the tripole grid are available. The one first introduced to CICE is called “U-fold”, which means that the poles and the grid cells between them are U cells on the grid. Alternatively the poles and the cells between them can be grid T cells, making a “T-fold.”

The files created by the UFS-UTILS cpld_gridgen utility, which reads the MOM6 ocean_hgrid.nc and writes out the CICE grid files and the mesh file show max nodecoord of +90.

The ocean domain that is created at runtime (looking at one of the ocean history files) shows a max geolat_c (corner) of +90.

Everything is consistent with the ice_in namelist setting tripole. The domain is folded along the U-row (CICE) or in MOM6 terms, the "Bu" points. The actual pole lies at a corner grid point.

If the pole was located on T-cells, you'd have a T-fold and the namelist setting would need to be tripoleT.

@DeniseWorthen
Thanks so much for this detail and checking it all! Glad that all is done "right"! 🙂

One thing is that there are

(zap_snow_temperature)zap_snow_temperature: temperature out of bounds! messages in the stdout.

Suggested fix to eliminate: you have both sea ice on land, and a bunch of corrupted sea ice values:
The other thing to check for is whether there is snow or ice on land in the restart file

if (sum(aicen) > 0 .and. kmt == 0) then sea ice exists on land; same for vicen and vsnon
Necessary check(s):
There are a bunch of 9.969E+36 aicen values on land along the left hand-size of the domain i<500 and a bunch of sea ice on land in the Barents Sea

This figure is sum(aicen) for kmt==0 (land) from the CICE restart in the wcoss2 run directory I initially got from Dan (iced.2025-05-01-64800.nc).

One problem area is in the far upper right corner where aice 0-1 exists on land.

Second issue are the vertical stripes on the left hand side (4.985E+37 = 5*9.969E+36). I think an there is an out-of-bounds array issue somewhere in the code which creates the restart. Has anyone tried running that code in debug?

@DeniseWorthen

Thanks @DeniseWorthen

Please note this @DanIredell-NOAA issue. Will have to be addressed .. sometime .. When we can get to it ..

Navigation Menu

Repository navigation

Set up model run on WCOSS #90

Goal:

Tasks:

Starting point:

Next steps:

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Set up model run on WCOSS #90

Description

Goal:

Tasks:

Starting point:

Next steps:

Activity

DeniseWorthen commented on Nov 21, 2025

sanAkel commented on Nov 21, 2025

DeniseWorthen commented on Nov 21, 2025

sanAkel commented on Nov 21, 2025

sanAkel commented on Nov 23, 2025

DeniseWorthen commented on Nov 23, 2025

DeniseWorthen commented on Nov 24, 2025

sanAkel commented on Nov 24, 2025

DeniseWorthen commented on Dec 10, 2025

sanAkel commented on Dec 10, 2025

DeniseWorthen commented on Dec 12, 2025

sanAkel commented on Dec 13, 2025

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions