XIOS indexing error in LFRic coupled climate model at lfric_apps vn3.2

[Dan Copsey (DanCopsey)]

When I run the LFRic coupled climate model for 1 month at lfric_apps vn3.2 I get the following XIOS error:

> Error [void CGrid::computeClientIndex()] : In file '/home/users/harry.shepherd/cylc-run/infrastructure_suite_deploy_exab_r5889/run1/share/xios/src/node/grid.cpp', line 735 -> Number of local index on client is different from number of received global indexRank of sent client 0.Number of local index 104. Number of received global index 119.

I have narrowed it down to be something to do with the checkpoint file as if I turn off checkpoint_write then it runs fine. However the failure seems to be at the start of the simulation so maybe it is XIOS setting things up. Running for 1 or 2 days also works fine so it is something to do with running for 1 month. I am running XIOS2.5 so maybe it is something to do with the XIOS version I am using.

Does anyone have any idea why I am getting this failure? What can I try to investigate further or fix it?

My workflow is u-di620_update_to_apps3.2 running on EXAB.

Thanks,
Dan.

[Thomas Bendall (tommbendall)]

I have two ideas...

Maybe it's related to this constraint: https://github.com/MetOffice/simulation-systems/wiki/2026.07.1#lfric_atm-rank--partition-count-constraint (but I think our C64 and C144 numbers probably satisfy this?)

Or do you need to add the W2H wind fields to the initial file, by replicating: https://github.com/MetOffice/lfric_apps/pull/470/changes#diff-9b38c2aa340cb2d34108845c1dab61f8721749bff55f8d937a2b4dc29830eb00

[iboutle]

The 15 field difference to me points to MetOffice/lfric_apps#505 (i.e. the 15 coupling fields that are manually added to the checkpoint file). Not sure if something changed to conflict with Tim's fix for the coupled checkpointing so that it's no longer working as expected?

[Dan Copsey (DanCopsey)]

I think I have figured it out. I ran with stochastic physics turned off and it worked. I think the code that went into vn3.2 that adds the stochastic physics seeds to the checkpoint file does not work with XIOS2.5 and as the coupled climate model uses XIOS2.5 that means I get the error that I have.

[Dan Copsey (DanCopsey)]

This change was: MetOffice/lfric_apps#383

[Steve Mullerworth (stevemullerworth)]

Why does it work when running for 1 or 2 days? Doesn't the checkpoint dump get written at the end of the run for any length run?

[Dan Copsey (DanCopsey)]

That was the clue because the 1 and 2 day runs had stochastic physics off whereas the 30 day run had it on. I initially thought it was the run length that was making the difference but it was the stochastic physics.

[Steve Mullerworth (stevemullerworth)]

This is surprising because it uses the same mechanisms to checkpoint the random seed which correctly appears in the dump of the working test.

I have checked your iodef file and there appear to be the correct references to the grids and axis for the stochastic physics arrays, so that bit of the upgrade seems to be fine.

I will try and turn on checkpointing and stochastic physics on in the lfric_coupled test to see if it is reproducible.

[Dan Copsey (DanCopsey)]

I have recreated the failure by turning on both stochastic physics and checkpointing in the lfric_coupled rose-stem test and running it. The changes I made to add these to the lfric_coupled rose stem test are in this branch:

https://github.com/DanCopsey/lfric_apps/tree/stochasic_checkpoint_rose_stem

I have made an lfric_apps issue in order to progress this further and get these turned on in the lfric_coupled rose-stem test alongside a fix:

MetOffice/lfric_apps#706