The 3dvar_O30kmIE60km_ColdStart test hangs in the Variational1 step.

[jim-p-w]

test/testinput/3dvar_O30kmIE60km_ColdStart.yaml hangs (and is killed by the PBS scheduler) when running the Variational1 step.
The hang is caused by an additional variable, refl10cm, being added to the definition of the da_state stream in MPAS-Model. The variational application hangs when trying to update the analysis file, which is generated by the mpas_atmosphere app in the ColdForecast step.
Note this doesn't happen when running the 3dvar_OIE120km_ColdStart.yaml test scenario. It appears to be related to using a hybrd 30km/60km resolution.

The underlying cause of this is unknown.

The hang goes away when building MPAS-Model with the refl10cm variable removed from the da_state stream definition. The problem also goes away when adding the Time variable to the definition of the da_state stream! The problem also went away when adding the variable rho_zz(Time, nCells, nVertLevels) to the da_state stream.

To Reproduce
Steps to reproduce the behavior:

1. Run the test/testinput/3dvar_O30kmIE60km_ColdStart.yaml scenario using the develop branch of MPAS-Workflow (which uses the binaries created with the develop branch of mpas-bundle). The variational1 step times out.

Expected behavior
This test should work.

Additional context
The hang occurs when using either the SMIOL I/O layer or the PIO I/O layer in MPAS-Model.

Changing the order of the variables in the da_state stream did nothing.

The MPAS-Model SMIOL code was instrumented with additional logging. These logs showed that all of the mpi ranks hung when making calls to ncmpi_enddef.

See the feature/da_state_add_Time branch of https://github.com/jim-p-w/MPAS-Model for an example of a workaround.

[byoung-joo]

@amstokely,

Another workaround you suggested is .. changing # of io tasks or io stride slightly by modifying namelist.atmosphere.

This will affect not only overwriting analysis file, but also reading the background (and potentially ensemble?) files.
I hope this would not affect the overall performance (i.e., runtime), for example, with 30km-60km or 15km-30km 3d-hybrid DA.

[jim-p-w]

@byoung-joo The stride setting we are using (stride=1) means every thread does it's own I/O to/from disk. It is more efficient to group writes. Michael has found via testing that a stride of 32 is an efficient setting for running a forecast on derecho.

A stride of 32 means threads are placed into groups of 32, and one thread in each group actual does the I/O. The other threads in the group send their I/O requests to the group's I/O thread. The I/O thread can consolidate reads or writes.

All of the cylc level 1 tests passed with that setting.
I will run a few cycles of a 30km-60km scenario to check performance.
I don't see an existing 3dhybrid scenario with 15-30m km resolution. Do you have one I can use?

[byoung-joo]

That sounds good. It would be nice if we can find a better IO configuration (either from others feedback or our own) in general!

15-30km setting is being used in our latest scientific experiment, based on release/3.0.3.
However, I don't know if we have that configuration working on top of develop.

[jim-p-w]

I have run the the tier 1 cylc tests, all of the run times are comparable, with the exception of the 3dvar_O30kmIE60km_ColdStart test, which hangs using stride of 1, and passes using stride of 32.
See this table for run time comparisons.

I am running the week long cylc simulation so we can check for differences in results between using stride 1 vs stride 32.

[byoung-joo]

I have run the the tier 1 cylc tests, all of the run times are comparable, with the exception of the 3dvar_O30kmIE60km_ColdStart test, which hangs using stride of 1, and passes using stride of 32.

This definitely sounds encouraging!

FYI: @snyderdad and @liujake,
Another workaround (by modifying the namelist.atmosphere setting) seems to be reasonably working.

[snyderdad]

If we have a workaround that will close this issue, it would be good to have an open issue in MPAS that documents the underlying behavior we're working around.

@amstokely would you do that please, if @mgduda agrees?

[jim-p-w]

The week long 3denvar simulation using stride=32 shows no differences from the 11/30 run which used stride=1:
See the 2025-12-11-3denvar row

[mgduda]

@amstokely and I are looking into a solution that will hopefully be more robust than changing the io_stride to 32 (though I would still advocate for increasing the stride purely for I/O performance reasons). Essentially, allocating extra space in the netCDF header should allow for future modifications of a file (e.g., adding new variables and attributes, and overwriting text attributes with longer values) without a need to re-allocate the header. The changes to allocate extra header space should require no more than half a dozen lines of new code in SMIOL.

@amstokely Could you open an issue in the MPAS-Dev/MPAS-Model repository detailing the I/O hang issue? I would guess that this can be described in a way that isn't overly MPAS-JEDI specific. We could do a few quick tests, but a confluence of several factors seems necessary in order to trigger a hang:

• Use of the PnetCDF to write to an existing file in parallel
• "Sufficiently large" number of I/O tasks writing to the file
• Writes to the file require reallocation of the netCDF header

[amstokely]

@amstokely and I are looking into a solution that will hopefully be more robust than changing the io_stride to 32 (though I would still advocate for increasing the stride purely for I/O performance reasons). Essentially, allocating extra space in the netCDF header should allow for future modifications of a file (e.g., adding new variables and attributes, and overwriting text attributes with longer values) without a need to re-allocate the header. The changes to allocate extra header space should require no more than half a dozen lines of new code in SMIOL.

@amstokely Could you open an issue in the MPAS-Dev/MPAS-Model repository detailing the I/O hang issue? I would guess that this can be described in a way that isn't overly MPAS-JEDI specific. We could do a few quick tests, but a confluence of several factors seems necessary in order to trigger a hang:

• Use of the PnetCDF to write to an existing file in parallel
• "Sufficiently large" number of I/O tasks writing to the file
• Writes to the file require reallocation of the netCDF header

@mgduda How is the description in issue 1385?