Lfric2lfric multi cpu

[ukmo-juan-castillo]

PR Summary

Sci/Tech Reviewer: @mo-rickywong
Code Reviewer: @TeranIvy

Change the type of the partitioning key instance member for lfric2lfric, so that lfric2lfric can run with more than one CPU

Tagging Iva and Steve so they are aware of the suspect output from lfric2lfric on multiple CPUs
@TeranIvy , @stevemullerworth

• closes lfric2lfric: run with multiple CPUs #192
• fixes lfric2lfric: run with multiple CPUs #192
• blocks Lfric2lfric read order #275 Lfric2lfric no multi partition #291

Code Quality Checklist

• I have performed a self-review of my own code
• My code follows the project's style guidelines
• Comments have been included that aid understanding and enhance the readability of the code
• My changes generate no new warnings
• All automated checks in the CI pipeline have completed successfully

Testing

• I have tested this change locally, using the LFRic Apps rose-stem suite
• If any tests fail (rose-stem or CI) the reason is understood and acceptable (e.g. kgo changes)
• I have added tests to cover new functionality as appropriate (e.g. system tests, unit tests, etc.)
• Any new tests have been assigned an appropriate amount of compute resource and have been allocated to an appropriate testing group (i.e. the developer tests are for jobs which use a small amount of compute resource and complete in a matter of minutes)

No other tests where performed, just modified existing tests so that the new functionality could be tested. Therefore, there were KGO changes that were addressed.

trac.log

Test Suite Results - lfric_apps - test_lfric2lfric_multi_CPU/run1

Suite Information

Item	Value
Suite Name	test_lfric2lfric_multi_CPU/run1
Suite User	juan.m.castillo
Workflow Start	2026-02-25T13:14:59
Groups Run	all

Dependency	Reference	Main Like
casim	MetOffice/casim@2025.12.1	True
jules	MetOffice/jules@69aaf4d	True
lfric_apps	ukmo-juan-castillo/lfric_apps@test_lfric2lfric_multi_CPU	False
lfric_core	MetOffice/lfric_core@2ed3e79	True
moci	MetOffice/moci@2025.12.1	True
SimSys_Scripts	MetOffice/SimSys_Scripts@2025.12.1	True
socrates	MetOffice/socrates@2025.12.1	True
socrates-spectral	MetOffice/socrates-spectral@2025.12.1	True
ukca	MetOffice/ukca@2025.12.1	True

Task Information

✅ succeeded tasks - 1515

Security Considerations

• I have reviewed my changes for potential security issues
• Sensitive data is properly handled (if applicable)
• Authentication and authorisation are properly implemented (if applicable)

Performance Impact

• Performance of the code has been considered and, if applicable, suitable performance measurements have been conducted

AI Assistance and Attribution

• Some of the content of this change has been produced with the assistance of Generative AI tool name (e.g., Met Office Github Copilot Enterprise, Github Copilot Personal, ChatGPT GPT-4, etc) and I have followed the Simulation Systems AI policy (including attribution labels)

Documentation

• Where appropriate I have updated documentation related to this change and confirmed that it builds correctly

PSyclone Approval

• If you have edited any PSyclone-related code (e.g. PSyKAl-lite, Kernel interface, optimisation scripts, LFRic data structure code) then please contact the TCD Team

Sci/Tech Review

• I understand this area of code and the changes being added
• The proposed changes correspond to the pull request description
• Documentation is sufficient (do documentation papers need updating)
• Sufficient testing has been completed

Please detail how you tested the functionality? i.e. what modifications did you make to test it? doesn't seem like the rose-stem test should pass given the partitioning checks with the presence of inter-grid maps in the input mesh file.

KGOs changes are numerous and not all insignificant, please provide plots/evidence that output from multiple cpus is not significantly different from runs with multiple cpus (plots?).

SciTech Review: Fail Suspect results from runs with multiple CPUs. While this change may enable lfric2lfric to run with multiple CPUs, the output maybe known but not necessarily good

Code Review

• All dependencies have been resolved
• Related Issues have been properly linked and addressed
• CLA compliance has been confirmed
• Code quality standards have been met
• Tests are adequate and have passed
• Documentation is complete and accurate
• Security considerations have been addressed
• Performance impact is acceptable

[mo-rickywong]

You need to add yourself to the contributors file (alphabetical order)

You say that you tested it to test that it works, how did you test it? As I recall you were using a input mesh file that had inter-grid maps. So did it not fail in the partitioning?

Looks like you just changed the number of mpis used on existing tasks.

The KGOs have changed, though I be suspicious of some of the checksums. There are a lot of differences, some only the last few digits, though some much larger. I think you need to provide some plots comparing 1 cpu vs multiple cpus to show that the fields are not significantly impacted.

[ukmo-juan-castillo]

You need to add yourself to the contributors file (alphabetical order)

This doesn't address the issue that there is no test-suite test that runs lfric2lfric on more than partition. Please create an issue to address it.

You say that you tested it to test that it works, how did you test it? As I recall you were using a input mesh file that had inter-grid maps. So did it not fail in the partitioning?

Please detail what you did to test you fix.

Added myself to the contributor list (added myself in a previous PR, but it has not been merged yet). I also fixed a detail in the new error message added to the code.

I modified the existing lfric2lfric rose stem tests, so that three of the jobs run with multiple CPUS: ral_seuk-C32_lam_MG, clim_gal9-C24_C12, and oasis_clim_gal9-C24_C12. That is the reason why there are KGO changes. I am afraid the results change slightly with the partitioning.

[mo-rickywong]

Suggested change

	help=In the lfric2lfric program, this variable indicates the
	=mesh for which the described partitioning is taking place.
	=
	=The only valid values of this variable in lfric2lfric
	=are 'source' to indicate the source grid, and 'destination'
	=to indicate the destination grid.
	help=The designation of the mesh to which this
	=partitioning profile is to be applied.
	=
	=Recognised designations are:
	= * 'source'
	= * 'destination'

[mo-rickywong]

Suggested change

	'Partitioning parameters for the source mesh were not found, '// &
	'please specify a partitioning namelist with mesh_target=source.'
	'Source mesh partitioning namelist (partitioning:source) not found.'

If the code cannot find the required partitioning profile namelist, then that is all it should state, the code doesn't know why it can't be found, so it cannot suggest a solution. For example, if this error message was given for Marks problem, he would have checked the namelist file which was not at fault. This message also would have to be updated if a different key instance member was used.

The Rose metadata is enough for the user to be able to check their configuration settings.

[mo-rickywong]

Suggested change

	'Partitioning parameters for the source mesh were not found, '// &
	'Destination mesh partitioning namelist (partitioning:destination) not found.'

As before, the code doesn't know why the namelist is not found. Also the error message still referred to the source mesh.

[mo-rickywong]

You need to add yourself to the contributors file (alphabetical order)
This doesn't address the issue that there is no test-suite test that runs lfric2lfric on more than partition. Please create an issue to address it.
You say that you tested it to test that it works, how did you test it? As I recall you were using a input mesh file that had inter-grid maps. So did it not fail in the partitioning?
Please detail what you did to test you fix.

Added myself to the contributor list (added myself in a previous PR, but it has not been merged yet). I also fixed a detail in the new error message added to the code.

I modified the existing lfric2lfric rose stem tests, so that three of the jobs run with multiple CPUS: ral_seuk-C32_lam_MG, clim_gal9-C24_C12, and oasis_clim_gal9-C24_C12. That is the reason why there are KGO changes. I am afraid the results change slightly with the partitioning.

I think you should edit the contributors list anyway, as it will block this change going on in the event that this PR is ready to commit first.

Please refer to my comments in the Scitech review. Please provide some evidence that there is not significant changes in the output (plots)

[ukmo-juan-castillo]

Thank you for the proposed changes, they make the code clearer.

The issue with the inter-grid maps will mainly affect tests regridding from a global to a regional mesh, and will be the purpose of a different issue. That is the reason why the added multi CPU tests in this issue are global to global and regional to regional.

I looked at the differences in timing between tests with single and multiple CPUs, and although the multiple CPU tests were slightly faster, the tests run so quickly that it is difficult to draw conclusions about performance. I recommend 'real case' tests as the ones Mark is doing to properly assess performance.

After that I looked at differences in results with the nctools. In the global to global test, the differences in the output comes from the longitude and latitude of grid faces and nodes, not from the processed fields. In the regional to regional tests, there are differences in the longitude and latitude of grid faces only, and also in some of the 2D fields defined as (vertical coordinate, horizontal grid). I show some plots for the variable 'area_fraction', which is one of the variables with differences when using a different number of CPUs (the scale is the same for all plots and go from 0 to 1).

Source field

Destination field using 1 CPU and the map regridding method

Destination field using 4 CPUs and the map regridding method

Destination field using 4 CPUs and the oasis regridding method

[mo-rickywong]

I wouldn't be concerned about speed at this juncture, My comments where based of the KGO differences and attribution of them. You haven't shown what the 1 CPU Oasis regridded method produces or is that not possible?

If the differences are in the long/lats please provide example magnitudes? or the differences?

[ukmo-juan-castillo]

I wouldn't be concerned about speed at this juncture, My comments where based of the KGO differences and attribution of them. You haven't shown what the 1 CPU Oasis regridded method produces or is that not possible?

If the differences are in the long/lats please provide example magnitudes? or the differences?

We looked in the past that the map and the oasis method gave the same results with one CPU, and this is still true although I did not include the plot. We find the same here with 4 CPUs.

I obtain lon/lat differences if I use ncdiff to generate the differences, but when I look at the actual longitude and latitude values in the lfric2lfric output using 1 or multiple CPUs, they are actually the same. I do not know why ncdiff behaves like this.

[mo-rickywong]

I wouldn't be concerned about speed at this juncture, My comments where based of the KGO differences and attribution of them. You haven't shown what the 1 CPU Oasis regridded method produces or is that not possible?
If the differences are in the long/lats please provide example magnitudes? or the differences?

We looked in the past that the map and the oasis method gave the same results with one CPU, and this is still true although I did not include the plot. We find the same here with 4 CPUs.

I obtain lon/lat differences if I use ncdiff to generate the differences, but when I look at the actual longitude and latitude values in the lfric2lfric output using 1 or multiple CPUs, they are actually the same. I do not know why ncdiff behaves like this.

ncdiff appears to checks differences at a lower tolerance than just dumping the output. Again, please post the differences in the lat lon values (a representative sample)

[ukmo-juan-castillo]

I wouldn't be concerned about speed at this juncture, My comments where based of the KGO differences and attribution of them. You haven't shown what the 1 CPU Oasis regridded method produces or is that not possible?
If the differences are in the long/lats please provide example magnitudes? or the differences?

We looked in the past that the map and the oasis method gave the same results with one CPU, and this is still true although I did not include the plot. We find the same here with 4 CPUs.
I obtain lon/lat differences if I use ncdiff to generate the differences, but when I look at the actual longitude and latitude values in the lfric2lfric output using 1 or multiple CPUs, they are actually the same. I do not know why ncdiff behaves like this.

ncdiff appears to checks differences at a lower tolerance than just dumping the output. Again, please post the differences in the lat lon values (a representative sample)

There are no differences in lat lon values when comparing the values directly. There are some differences when comparing using ncdiff. I am confident that the lat lon values is not the problem. I suspect the problem is how lfric reads different types of data, I am preparing a test to check this theory.

[mo-rickywong]

As requested, please post the differences in the values you have from ncdiff.

[ukmo-juan-castillo]

As requested, please post the differences in the values you have from ncdiff.

Taken directly from ncdiff:

Mesh2d_face_x = -0.7626276, -0.6977054, -0.6327863, -0.5678701, -0.502957,
-0.4380471, -0.3731405, -0.3082374, -0.2433378, -0.1784418, -0.1135496,
-0.04866117, 0.01622329, 0.08110369, 0.1459799, 0.210852, 0.2757196,
0.3405828, 0.4054414, 0.4702953, 0.5351445, 0.5999888, 0.6648281,
0.7296623, 0.7944914, 0.8593151, 0.9241334, 0.9889463, 1.053753,
1.118555, 1.183351, 1.248141, -0.7637985, -0.69892, -0.6340445,
-0.569172, -0.5043026, -0.4394364, -0.3745736, -0.3097141, -0.2448581,
-0.1800058, -0.1151572, -0.05031246, 0.01452837, 0.07936516, 0.1441978,
0.2090262, 0.2738502, 0.3386698, 0.4034849, 0.4682952, 0.5331009,
0.5979016, 0.6626974, 0.7274881, 0.7922736, 0.8570539, 0.9218287,
0.9865981, 1.051362, 1.11612, 1.180872, 1.245619, -0.7649677, -0.700133,
-0.6353011, -0.5704722, -0.5056465, -0.4408239, -0.3760047, -0.3111888,
-0.2463765, -0.1815677, -0.1167627, -0.05196154, 0.01283571, 0.07762894,
0.142418, 0.2072029, 0.2719834, 0.3367595, 0.401531, 0.4662979, 0.53106,
0.5958173, 0.6605695, 0.7253168, 0.7900589, 0.8547958, 0.9195272,
0.9842532, 1.048974, 1.113688, 1.178397, 1.2431, -0.7661355, -0.7013443,
-0.636556, -0.5717707, -0.5069885, -0.4422095, -0.3774339, -0.3126616,
-0.2478928, -0.1831276, -0.1183661, -0.05360843, 0.01114531, 0.07589503,
0.1406406, 0.205382, 0.270119, 0.3348517, 0.3995797, 0.4643032,
0.5290219, 0.5937357, 0.6584446, 0.7231484, 0.7878471, 0.8525406,
0.9172286, 0.9819112, 1.046588, 1.11126, 1.175925, 1.240585, -0.7673016,
-0.702554, -0.6378093, -0.5730675, -0.5083289, -0.4435934, -0.3788612,
-0.3141323, -0.249407, -0.1846853, -0.1199673, -0.05525312, 0.009457169,
0.07416344, 0.1388656, 0.2035636, 0.2682572, 0.3329464, 0.397631,
0.4623111, 0.5269864, 0.5916569, 0.6563224, 0.7209829, 0.7856383,
0.8502884, 0.9149331, 0.9795725, 1.044206, 1.108834, 1.173457, 1.238073,
-0.7684662, -0.7037621, -0.6390609, -0.5743625, -0.5096673, -0.4449753,
-0.3802865, -0.3156012, -0.2509193, -0.186241, -0.1215664, -0.05689562,
0.007771273, 0.07243416, 0.1370929, 0.2017475, 0.2663978, 0.3310436,
0.395685, 0.4603217, 0.5249537, 0.5895808, 0.6542031, 0.7188203,
0.7834324, 0.8480392, 0.9126408, 0.9772368, 1.041827, 1.106412, 1.170991,
1.235565, -0.7696293, -0.7049686, -0.6403108, -0.5756559, -0.511004,
-0.4463554, -0.38171, -0.317068, -0.2524295, -0.1877946, -0.1231634,
-0.05853593, 0.006087618, 0.07070718, 0.1353226, 0.1999339, 0.2645409,
0.3291434, 0.3937415, 0.4583349, 0.5229236, 0.5875075, 0.6520866,
0.7166606, 0.7812294, 0.8457931, 0.9103514, 0.9749043, 1.039452,
1.103993, 1.168529, 1.23306, -0.7707908, -0.7061735, -0.641559,
-0.5769475, -0.512339, -0.4477337, -0.3831316, -0.318533, -0.2539378,
-0.1893462, -0.1247582, -0.06017407, 0.0044062, 0.06898249, 0.1335547,
0.1981227, 0.2626864, 0.3272457, 0.3918006, 0.4563508, 0.5208963,
0.585437, 0.6499729, 0.7145036, 0.7790294, 0.8435498, 0.908065,
0.9725748, 1.037079, 1.101578, 1.166071, 1.230558, -0.7719508,
-0.7073768, -0.6428056, -0.5782373, -0.5136721, -0.4491101, -0.3845513,
-0.3199959, -0.255444, -0.1908956, -0.126351, -0.06181003, 0.002727014,
0.06726009, 0.1317891, 0.1963139, 0.2608344, 0.3253506, 0.3898622,
0.4543693, 0.5188717, 0.5833693, 0.647862, 0.7123497, 0.7768322,
0.8413096, 0.9057817, 0.9702485, 1.03471, 1.099165, 1.163615, 1.228059,
-0.7731093, -0.7085785, -0.6440505, -0.5795255, -0.5150035, -0.4504847,
-0.3859692, -0.321457, -0.2569483, -0.1924431, -0.1279416, -0.06344382,
0.001050053, 0.06553996, 0.1300258, 0.1945075, 0.2589849, 0.3234579,
0.3879264, 0.4523904, 0.5168497, 0.5813042, 0.6457538, 0.7101985,
0.774638, 0.8390723, 0.9035015, 0.9679251, 1.032343, 1.096756, 1.161163,
1.225564, -0.7742661, -0.7097785, -0.6452938, -0.5808119, -0.5163332,
-0.4518575, -0.3873851, -0.3229161, -0.2584505, -0.1939885, -0.1295301,
-0.06507546, -0.0006246865, 0.06382212, 0.1282649, 0.1927035, 0.2571378,
0.3215677, 0.3859932, 0.4504142, 0.5148304, 0.5792419, 0.6436485,
0.7080501, 0.7724467, 0.8368381, 0.9012242, 0.9656049, 1.02998, 1.09435,
1.158714, 1.223072, -0.7754215, -0.710977, -0.6465354, -0.5820967,
-0.517661, -0.4532285, -0.3887992, -0.3243733, -0.2599508, -0.1955318,
-0.1311165, -0.06670493, -0.002297211, 0.06210655, 0.1265063, 0.1909018,
0.2552931, 0.3196801, 0.3840626, 0.4484405, 0.5128137, 0.5771823,
0.641546, 0.7059046, 0.7702582, 0.8346067, 0.8989499, 0.9632878, 1.02762,
1.091947, 1.156268, 1.220583, -0.7765754, -0.7121739, -0.6477754,
-0.5833797, -0.5189871, -0.4545977, -0.3902114, -0.3258285, -0.261449,
-0.1970731, -0.1327008, -0.06833224, -0.003967525, 0.06039325, 0.12475,
0.1891026, 0.2534509, 0.3177949, 0.3821345, 0.4464695, 0.5107998,
0.5751254, 0.6394462, 0.703762, 0.7680727, 0.8323783, 0.8966787,
0.9609737, 1.025263, 1.089547, 1.153826, 1.218098, -0.7777277,
-0.7133693, -0.6490138, -0.5846611, -0.5203115, -0.455965, -0.3916218,
-0.3272818, -0.2629453, -0.1986124, -0.134283, -0.06995741, -0.005635635,
0.05868221, 0.122996, 0.1873057, 0.2516111, 0.3159122, 0.3802089,
0.444501, 0.5087885, 0.5730712, 0.6373492, 0.7016221, 0.7658901,
0.8301529, 0.8944104, 0.9586626, 1.022909, 1.087151, 1.151386, 1.215616,
-0.7788785, -0.7145631, -0.6502505, -0.5859408, -0.5216341, -0.4573306,
-0.3930303, -0.3287332, -0.2644397, -0.2001496, -0.1358632, -0.07158044,
-0.007301545, 0.05697342, 0.1212444, 0.1855112, 0.2497738, 0.314032,
0.3782859, 0.4425352, 0.5067798, 0.5710198, 0.6352549, 0.6994851,
0.7637103, 0.8279303, 0.8921451, 0.9563546, 1.020559, 1.084757, 1.14895,
1.213137, -0.7800277, -0.7157552, -0.6514856, -0.5872188, -0.522955,
-0.4586943, -0.3944369, -0.3301827, -0.265932, -0.2016848, -0.1374412,
-0.07320133, -0.008965261, 0.05526689, 0.119495, 0.183719, 0.2479388,
0.3121543, 0.3763654, 0.4405719, 0.5047739, 0.568971, 0.6331635,
0.6973509, 0.7615334, 0.8257107, 0.8898828, 0.9540496, 1.018211,
1.082367, 1.146517, 1.210662, -0.7811755, -0.7169459, -0.652719,
-0.5884951, -0.5242741, -0.4600563, -0.3958417, -0.3316303, -0.2674224,
-0.203218, -0.1390172, -0.07482009, -0.01062679, 0.0535626, 0.117748,
0.1819292, 0.2461063, 0.3102791, 0.3744474, 0.4386112, 0.5027705,
0.566925, 0.6310747, 0.6952195, 0.7593593, 0.823494, 0.8876235,
0.9517477, 1.015866, 1.07998, 1.144087, 1.208189, -0.7823218, -0.7181349,
-0.6539509, -0.5897697, -0.5255916, -0.4614165, -0.3972446, -0.333076,
-0.2689109, -0.2047492, -0.1405911, -0.07643671, -0.01228613, 0.05186055,
0.1160032, 0.1801418, 0.2442762, 0.3084063, 0.372532, 0.4366532,
0.5007697, 0.5648816, 0.6289887, 0.6930909, 0.7571881, 0.8212802,
0.8853672, 0.9494488, 1.013525, 1.077596, 1.141661, 1.20572, -0.7834665,
-0.7193224, -0.6551811, -0.5910426, -0.5269072, -0.4627748, -0.3986457,
-0.3345198, -0.2703973, -0.2062783, -0.1421629, -0.07805122, -0.0139433,
0.05016074, 0.1142608, 0.1783567, 0.2424485, 0.306536, 0.370619,
0.4346976, 0.4987716, 0.5628409, 0.6269055, 0.6909651, 0.7550198,
0.8190693, 0.8831137, 0.9471529, 1.011187, 1.075215, 1.139237, 1.203254,
-0.7846098, -0.7205083, -0.6564097, -0.5923139, -0.5282211, -0.4641314,
-0.4000449, -0.3359617, -0.2718818, -0.2078055, -0.1437327, -0.0796636,
-0.01559829, 0.04846315, 0.1125206, 0.176574, 0.2406232, 0.3046681,
0.3687086, 0.4327447, 0.4967761, 0.5608029, 0.6248249, 0.6888421,
0.7528543, 0.8168613, 0.8808633, 0.9448599, 1.008851, 1.072837, 1.136817,
1.200792, -0.7857515, -0.7216927, -0.6576367, -0.5935835, -0.5295334,
-0.4654863, -0.4014423, -0.3374017, -0.2733644, -0.2093306, -0.1453004,
-0.08127388, -0.01725111, 0.04676779, 0.1107827, 0.1747936, 0.2388002,
0.3028027, 0.3668007, 0.4307942, 0.4947833, 0.5587676, 0.6227471,
0.6867219, 0.7506916, 0.8146563, 0.8786158, 0.94257, 1.006519, 1.070462,
1.1344, 1.198332, -0.7868918, -0.7228755, -0.6588621, -0.5948515,
-0.5308439, -0.4668393, -0.4028379, -0.3388398, -0.2748451, -0.2108538,
-0.1468661, -0.08288205, -0.01890177, 0.04507464, 0.1090471, 0.1730155,
0.2369797, 0.3009396, 0.3648953, 0.4288464, 0.492793, 0.5567349,
0.620672, 0.6846044, 0.7485318, 0.812454, 0.8763712, 0.9402831, 1.00419,
1.068091, 1.131986, 1.195876, -0.7880306, -0.7240568, -0.6600859,
-0.5961178, -0.5321527, -0.4681906, -0.4042317, -0.3402761, -0.2763238,
-0.212375, -0.1484297, -0.08448811, -0.02055027, 0.04338371, 0.1073137,
0.1712397, 0.2351615, 0.2990791, 0.3629923, 0.4269011, 0.4908053,
0.5547048, 0.6185997, 0.6824897, 0.7463747, 0.8102547, 0.8741296,
0.9379992, 1.001863, 1.065722, 1.129576, 1.193423, -0.7891678,
-0.7252365, -0.6613081, -0.5973824, -0.5334597, -0.4695401, -0.4056236,
-0.3417104, -0.2778006, -0.2138941, -0.1499913, -0.08609208, -0.02219662,
0.04169499, 0.1055826, 0.1694663, 0.2333457, 0.297221, 0.3610919,
0.4249583, 0.4888202, 0.5526775, 0.61653, 0.6803777, 0.7442205,
0.8080583, 0.8718908, 0.9357182, 0.9995403, 1.063357, 1.127168, 1.190974,
-0.7903036, -0.7264147, -0.6625286, -0.5986454, -0.5347651, -0.4708879,
-0.4070137, -0.3431429, -0.2792754, -0.2154113, -0.1515508, -0.08769395,
-0.02384082, 0.04000847, 0.1038538, 0.1676951, 0.2315323, 0.2953653,
0.3591939, 0.4230181, 0.4868377, 0.5506527, 0.614463, 0.6782686,
0.7420691, 0.8058646, 0.8696551, 0.9334403, 0.9972202, 1.060995,
1.124764, 1.188527, -0.7914379, -0.7275914, -0.6637477, -0.5999067,
-0.5360688, -0.4722339, -0.4084021, -0.3445735, -0.2807483, -0.2169266,
-0.1531083, -0.08929374, -0.02548288, 0.03832414, 0.1021272, 0.1659263,
0.2297212, 0.293512, 0.3572984, 0.4210804, 0.4848578, 0.5486307,
0.6123987, 0.6761621, 0.7399205, 0.8036739, 0.8674222, 0.9311653,
0.994903, 1.058635, 1.122362, 1.186084, -0.7925708, -0.7287666,
-0.6649651, -0.6011664, -0.5373707, -0.4735781, -0.4097886, -0.3460023,
-0.2822193, -0.2184398, -0.1546638, -0.09089145, -0.0271228, 0.03664201,
0.1004029, 0.1641598, 0.2279125, 0.2916611, 0.3554053, 0.4191451,
0.4828805, 0.5466112, 0.6103372, 0.6740584, 0.7377747, 0.801486,
0.8651922, 0.9288932, 0.9925889, 1.056279, 1.119964, 1.183643,
-0.7937021, -0.7299401, -0.6661809, -0.6024245, -0.538671, -0.4749206,
-0.4111733, -0.3474292, -0.2836884, -0.2199511, -0.1562173, -0.09248707,
-0.02876059, 0.03496207, 0.09868082, 0.1623956, 0.2261062, 0.2898126,
0.3535148, 0.4172125, 0.4809057, 0.5445943, 0.6082783, 0.6719574,
0.7356316, 0.7993009, 0.8629651, 0.9266241, 0.9902778, 1.053926,
1.117569, 1.181206, -0.794832, -0.7311122, -0.6673952, -0.6036809,
-0.5399696, -0.4762613, -0.4125562, -0.3488542, -0.2851556, -0.2214604,
-0.1577687, -0.09408063, -0.03039626, 0.03328431, 0.09696097, 0.1606336,
0.2243022, 0.2879665, 0.3516266, 0.4152823, 0.4789335, 0.5425801,
0.606222, 0.6698592, 0.7334914, 0.7971187, 0.8607409, 0.924358,
0.9879697, 1.051576, 1.115177, 1.178772, -0.7959604, -0.7322828,
-0.6686078, -0.6049357, -0.5412665, -0.4776003, -0.4139373, -0.3502774,
-0.2866209, -0.2229678, -0.1593181, -0.09567212, -0.0320298, 0.03160873,
0.09524335, 0.158874, 0.2225005, 0.2861229, 0.349741, 0.4133546,
0.4769638, 0.5405685, 0.6041684, 0.6677636, 0.7313539, 0.7949393,
0.8585196, 0.9220947, 0.9856646, 1.049229, 1.112788, 1.176342,
-0.7970874, -0.7334518, -0.6698189, -0.6061889, -0.5425617, -0.4789376,
-0.4153166, -0.3516988, -0.2880843, -0.2244732, -0.1608656, -0.09726154,
-0.03366122, 0.02993531, 0.09352796, 0.1571166, 0.2207012, 0.2842816,
0.3478577, 0.4114295, 0.4749967, 0.5385594, 0.6021175, 0.6656708,
0.7292192, 0.7927627, 0.8563011, 0.9198344, 0.9833624, 1.046885,
1.110402, 1.173914, -0.7982129, -0.7346193, -0.6710284, -0.6074404,
-0.5438552, -0.4802732, -0.4166941, -0.3531183, -0.2895457, -0.2259766,
-0.162411, -0.0988489, -0.03529053, 0.02826407, 0.09181478, 0.1553615,
0.2189042, 0.2824427, 0.3459769, 0.4095068, 0.4730322, 0.536553,
0.6000692, 0.6635807, 0.7270873, 0.7905889, 0.8540856, 0.917577,
0.9810633, 1.044544, 1.10802, 1.17149 ;
}

Mesh2d_face_y = 51.98213, 51.98115, 51.98013, 51.97908, 51.97799, 51.97687,
51.97571, 51.97451, 51.97328, 51.97202, 51.97071, 51.96937, 51.968,
51.96659, 51.96514, 51.96366, 51.96214, 51.96058, 51.95899, 51.95737,
51.9557, 51.95401, 51.95227, 51.9505, 51.9487, 51.94685, 51.94498,
51.94306, 51.94111, 51.93913, 51.93711, 51.93505, 51.95214, 51.95116,
51.95014, 51.94909, 51.94801, 51.94688, 51.94572, 51.94453, 51.9433,
51.94203, 51.94073, 51.93939, 51.93801, 51.9366, 51.93516, 51.93368,
51.93216, 51.93061, 51.92902, 51.92739, 51.92573, 51.92403, 51.9223,
51.92053, 51.91873, 51.91689, 51.91501, 51.9131, 51.91115, 51.90916,
51.90715, 51.90509, 51.92215, 51.92117, 51.92015, 51.9191, 51.91801,
51.91689, 51.91574, 51.91454, 51.91331, 51.91204, 51.91074, 51.90941,
51.90803, 51.90662, 51.90518, 51.9037, 51.90218, 51.90063, 51.89904,
51.89742, 51.89576, 51.89406, 51.89233, 51.89056, 51.88876, 51.88692,
51.88504, 51.88313, 51.88119, 51.8792, 51.87719, 51.87513, 51.89215,
51.89118, 51.89016, 51.88911, 51.88803, 51.88691, 51.88575, 51.88456,
51.88333, 51.88206, 51.88076, 51.87943, 51.87805, 51.87664, 51.8752,
51.87372, 51.8722, 51.87065, 51.86906, 51.86744, 51.86578, 51.86409,
51.86236, 51.86059, 51.85879, 51.85695, 51.85508, 51.85317, 51.85122,
51.84924, 51.84723, 51.84517, 51.86216, 51.86119, 51.86017, 51.85912,
51.85804, 51.85692, 51.85576, 51.85457, 51.85334, 51.85208, 51.85078,
51.84944, 51.84807, 51.84666, 51.84522, 51.84374, 51.84222, 51.84068,
51.83909, 51.83747, 51.83581, 51.83411, 51.83239, 51.83062, 51.82882,
51.82698, 51.82511, 51.8232, 51.82126, 51.81928, 51.81726, 51.81521,
51.83217, 51.8312, 51.83018, 51.82914, 51.82805, 51.82693, 51.82578,
51.82458, 51.82336, 51.82209, 51.82079, 51.81946, 51.81809, 51.81668,
51.81524, 51.81376, 51.81225, 51.8107, 51.80911, 51.80749, 51.80584,
51.80414, 51.80241, 51.80065, 51.79885, 51.79702, 51.79514, 51.79324,
51.79129, 51.78931, 51.7873, 51.78525, 51.80218, 51.8012, 51.80019,
51.79914, 51.79806, 51.79694, 51.79579, 51.7946, 51.79337, 51.79211,
51.79081, 51.78947, 51.78811, 51.7867, 51.78526, 51.78378, 51.78227,
51.78072, 51.77914, 51.77752, 51.77586, 51.77417, 51.77244, 51.77068,
51.76888, 51.76705, 51.76518, 51.76327, 51.76133, 51.75935, 51.75734,
51.75529, 51.77219, 51.77121, 51.7702, 51.76916, 51.76807, 51.76696,
51.7658, 51.76461, 51.76339, 51.76212, 51.76083, 51.75949, 51.75813,
51.75672, 51.75528, 51.7538, 51.75229, 51.75074, 51.74916, 51.74754,
51.74589, 51.7442, 51.74247, 51.74071, 51.73891, 51.73708, 51.73521,
51.73331, 51.73137, 51.72939, 51.72738, 51.72533, 51.7422, 51.74122,
51.74021, 51.73917, 51.73808, 51.73697, 51.73581, 51.73462, 51.7334,
51.73214, 51.73084, 51.72951, 51.72814, 51.72674, 51.7253, 51.72382,
51.72231, 51.72077, 51.71918, 51.71757, 51.71591, 51.71423, 51.7125,
51.71074, 51.70894, 51.70711, 51.70524, 51.70334, 51.7014, 51.69943,
51.69742, 51.69537, 51.71221, 51.71123, 51.71022, 51.70918, 51.7081,
51.70698, 51.70583, 51.70464, 51.70341, 51.70215, 51.70086, 51.69953,
51.69816, 51.69676, 51.69532, 51.69384, 51.69233, 51.69079, 51.68921,
51.68759, 51.68594, 51.68425, 51.68253, 51.68077, 51.67897, 51.67714,
51.67528, 51.67337, 51.67144, 51.66946, 51.66746, 51.66541, 51.68222,
51.68124, 51.68023, 51.67919, 51.67811, 51.67699, 51.67584, 51.67465,
51.67343, 51.67217, 51.67088, 51.66954, 51.66818, 51.66677, 51.66534,
51.66386, 51.66236, 51.66081, 51.65923, 51.65762, 51.65597, 51.65428,
51.65256, 51.6508, 51.64901, 51.64717, 51.64531, 51.64341, 51.64148,
51.6395, 51.63749, 51.63545, 51.65223, 51.65125, 51.65024, 51.6492,
51.64812, 51.647, 51.64585, 51.64466, 51.64344, 51.64219, 51.64089,
51.63956, 51.6382, 51.6368, 51.63536, 51.63389, 51.63238, 51.63084,
51.62926, 51.62764, 51.62599, 51.62431, 51.62259, 51.62083, 51.61903,
51.61721, 51.61535, 51.61345, 51.61151, 51.60954, 51.60753, 51.60549,
51.62223, 51.62126, 51.62025, 51.61921, 51.61813, 51.61702, 51.61586,
51.61468, 51.61346, 51.6122, 51.6109, 51.60958, 51.60822, 51.60682,
51.60538, 51.60391, 51.6024, 51.60086, 51.59928, 51.59767, 51.59602,
51.59433, 51.59261, 51.59086, 51.58907, 51.58724, 51.58538, 51.58348,
51.58155, 51.57957, 51.57757, 51.57553, 51.59224, 51.59127, 51.59026,
51.58922, 51.58814, 51.58703, 51.58588, 51.58469, 51.58347, 51.58221,
51.58092, 51.57959, 51.57823, 51.57683, 51.5754, 51.57393, 51.57242,
51.57088, 51.56931, 51.56769, 51.56604, 51.56436, 51.56264, 51.56089,
51.5591, 51.55727, 51.55541, 51.55351, 51.55158, 51.54961, 51.54761,
51.54557, 51.56225, 51.56128, 51.56027, 51.55923, 51.55815, 51.55704,
51.55589, 51.55471, 51.55349, 51.55223, 51.55094, 51.54961, 51.54825,
51.54685, 51.54542, 51.54395, 51.54244, 51.5409, 51.53933, 51.53772,
51.53607, 51.53439, 51.53267, 51.53092, 51.52913, 51.5273, 51.52544,
51.52355, 51.52162, 51.51965, 51.51765, 51.51561, 51.53226, 51.53129,
51.53028, 51.52924, 51.52816, 51.52705, 51.5259, 51.52472, 51.5235,
51.52224, 51.52095, 51.51963, 51.51827, 51.51687, 51.51544, 51.51397,
51.51247, 51.51093, 51.50935, 51.50774, 51.5061, 51.50441, 51.5027,
51.50095, 51.49916, 51.49733, 51.49548, 51.49358, 51.49165, 51.48969,
51.48769, 51.48565, 51.50227, 51.5013, 51.50029, 51.49925, 51.49818,
51.49706, 51.49591, 51.49473, 51.49352, 51.49226, 51.49097, 51.48965,
51.48829, 51.48689, 51.48546, 51.48399, 51.48249, 51.48095, 51.47937,
51.47777, 51.47612, 51.47444, 51.47272, 51.47097, 51.46919, 51.46737,
51.46551, 51.46362, 51.46169, 51.45972, 51.45773, 51.45569, 51.47227,
51.47131, 51.4703, 51.46926, 51.46819, 51.46708, 51.46593, 51.46474,
51.46353, 51.46228, 51.46099, 51.45966, 51.4583, 51.45691, 51.45548,
51.45401, 51.45251, 51.45097, 51.4494, 51.44779, 51.44615, 51.44447,
51.44275, 51.44101, 51.43922, 51.4374, 51.43554, 51.43365, 51.43172,
51.42976, 51.42776, 51.42573, 51.44228, 51.44131, 51.44031, 51.43927,
51.4382, 51.43709, 51.43594, 51.43476, 51.43354, 51.43229, 51.431,
51.42968, 51.42832, 51.42693, 51.4255, 51.42403, 51.42253, 51.42099,
51.41942, 51.41782, 51.41617, 51.4145, 51.41278, 51.41103, 51.40925,
51.40743, 51.40557, 51.40369, 51.40176, 51.3998, 51.3978, 51.39577,
51.41229, 51.41132, 51.41032, 51.40928, 51.40821, 51.4071, 51.40595,
51.40477, 51.40356, 51.40231, 51.40102, 51.3997, 51.39834, 51.39695,
51.39552, 51.39405, 51.39255, 51.39102, 51.38945, 51.38784, 51.3862,
51.38452, 51.38281, 51.38106, 51.37928, 51.37746, 51.37561, 51.37372,
51.3718, 51.36983, 51.36784, 51.36581, 51.3823, 51.38133, 51.38033,
51.37929, 51.37822, 51.37711, 51.37597, 51.37479, 51.37357, 51.37232,
51.37104, 51.36971, 51.36835, 51.36696, 51.36554, 51.36407, 51.36258,
51.36104, 51.35947, 51.35786, 51.35622, 51.35455, 51.35284, 51.35109,
51.34931, 51.34749, 51.34564, 51.34375, 51.34183, 51.33987, 51.33788,
51.33585, 51.35231, 51.35134, 51.35034, 51.3493, 51.34823, 51.34712,
51.34598, 51.3448, 51.34359, 51.34233, 51.34105, 51.33973, 51.33838,
51.33698, 51.33556, 51.3341, 51.3326, 51.33106, 51.32949, 51.32789,
51.32625, 51.32458, 51.32287, 51.32112, 51.31934, 51.31752, 51.31567,
51.31379, 51.31187, 51.30991, 51.30791, 51.30589, 51.32232, 51.32135,
51.32035, 51.31931, 51.31824, 51.31713, 51.31599, 51.31481, 51.3136,
51.31235, 51.31107, 51.30975, 51.30839, 51.307, 51.30558, 51.30412,
51.30262, 51.30109, 51.29952, 51.29792, 51.29628, 51.29461, 51.2929,
51.29115, 51.28937, 51.28756, 51.28571, 51.28382, 51.2819, 51.27995,
51.27795, 51.27592, 51.29232, 51.29136, 51.29036, 51.28933, 51.28825,
51.28715, 51.286, 51.28483, 51.28362, 51.28237, 51.28108, 51.27976,
51.27841, 51.27702, 51.27559, 51.27414, 51.27264, 51.27111, 51.26954,
51.26794, 51.2663, 51.26463, 51.26292, 51.26118, 51.2594, 51.25759,
51.25574, 51.25386, 51.25193, 51.24998, 51.24799, 51.24596, 51.26233,
51.26137, 51.26037, 51.25933, 51.25826, 51.25716, 51.25602, 51.25484,
51.25363, 51.25238, 51.2511, 51.24978, 51.24843, 51.24704, 51.24561,
51.24416, 51.24266, 51.24113, 51.23956, 51.23796, 51.23633, 51.23466,
51.23295, 51.23121, 51.22943, 51.22762, 51.22577, 51.22389, 51.22197,
51.22002, 51.21803, 51.216, 51.23234, 51.23138, 51.23038, 51.22934,
51.22828, 51.22717, 51.22603, 51.22485, 51.22364, 51.2224, 51.22112,
51.2198, 51.21844, 51.21706, 51.21563, 51.21418, 51.21268, 51.21115,
51.20959, 51.20799, 51.20635, 51.20469, 51.20298, 51.20124, 51.19946,
51.19765, 51.1958, 51.19392, 51.19201, 51.19005, 51.18806, 51.18604,
51.20235, 51.20139, 51.20039, 51.19936, 51.19829, 51.19718, 51.19604,
51.19487, 51.19366, 51.19241, 51.19113, 51.18982, 51.18846, 51.18708,
51.18565, 51.1842, 51.1827, 51.18118, 51.17961, 51.17801, 51.17638,
51.17471, 51.17301, 51.17127, 51.16949, 51.16768, 51.16584, 51.16396,
51.16204, 51.16009, 51.1581, 51.15608, 51.17236, 51.1714, 51.1704,
51.16936, 51.1683, 51.16719, 51.16605, 51.16488, 51.16367, 51.16243,
51.16114, 51.15983, 51.15848, 51.15709, 51.15567, 51.15422, 51.15273,
51.1512, 51.14964, 51.14804, 51.1464, 51.14474, 51.14304, 51.1413,
51.13952, 51.13771, 51.13587, 51.13399, 51.13208, 51.13013, 51.12814,
51.12612, 51.14237, 51.14141, 51.14041, 51.13938, 51.13831, 51.13721,
51.13607, 51.1349, 51.13369, 51.13244, 51.13116, 51.12985, 51.1285,
51.12711, 51.12569, 51.12424, 51.12275, 51.12122, 51.11966, 51.11806,
51.11643, 51.11477, 51.11306, 51.11132, 51.10955, 51.10775, 51.1059,
51.10402, 51.10211, 51.10016, 51.09818, 51.09616, 51.11238, 51.11142,
51.11042, 51.10939, 51.10832, 51.10722, 51.10608, 51.10491, 51.1037,
51.10246, 51.10118, 51.09986, 51.09851, 51.09713, 51.09571, 51.09426,
51.09277, 51.09124, 51.08968, 51.08809, 51.08646, 51.08479, 51.08309,
51.08135, 51.07958, 51.07778, 51.07594, 51.07406, 51.07215, 51.0702,
51.06822, 51.0662, 51.08239, 51.08142, 51.08043, 51.0794, 51.07833,
51.07723, 51.07609, 51.07492, 51.07371, 51.07247, 51.07119, 51.06988,
51.06853, 51.06715, 51.06573, 51.06428, 51.06279, 51.06126, 51.0597,
51.05811, 51.05648, 51.05482, 51.05312, 51.05138, 51.04961, 51.04781,
51.04597, 51.04409, 51.04218, 51.04023, 51.03825, 51.03624, 51.05239,
51.05143, 51.05044, 51.04941, 51.04834, 51.04724, 51.0461, 51.04493,
51.04373, 51.04248, 51.04121, 51.0399, 51.03855, 51.03717, 51.03575,
51.0343, 51.03281, 51.03129, 51.02973, 51.02814, 51.02651, 51.02485,
51.02314, 51.02141, 51.01964, 51.01784, 51.016, 51.01413, 51.01221,
51.01027, 51.00829, 51.00628 ;
}

[mo-rickywong]

No, that's not it, have you used nccmp? I want to see the differences in the lat lon values in the output. and similarly in the global mesh files you are using as input.

[ukmo-juan-castillo]

No, that's not it, have you used nccmp? I want to see the differences in the lat lon values in the output. and similarly in the global mesh files you are using as input.

So far I only used ncdiff. Differences with nccmp are, for the regional model:

Variable Group Count Sum AbsSum Min Max Range Mean StdDev
checkpoint_Wtheta / 1 -1 1 -1 -1 0 -1 0
checkpoint_W3 / 1 -1.11253e+09 1.11253e+09 -1.11253e+09 -1.11253e+09 0 -1.11253e+09 0
checkpoint_W2 / 1 -1.11253e+09 1.11253e+09 -1.11253e+09 -1.11253e+09 0 -1.11253e+09 0
checkpoint_W2_node_x / 220544 0 145292 -1.98049 2.10612 4.0866 0 0.830915
checkpoint_W2_node_y / 220544 0 103486 -0.705284 0.823914 1.5292 0 0.500886
Mesh2d / 1 -1.11253e+09 1.11253e+09 -1.11253e+09 -1.11253e+09 0 -1.11253e+09 0
Mesh2d_edge_nodes / 192 0 192 -1 1 2 0 1.00261
wvar / 72704 0.00111539 6164.43 -2.45019 2.51847 4.96866 1.53416e-08 0.228017
leaf_area_index / 5062 -inf inf -1.79769e+308 1.79769e+308 inf -inf -nan
canopy_height / 5061 -inf inf -1.79769e+308 1.79769e+308 inf -inf -nan
tile_fraction / 8897 0.000115625 1078.55 -0.855488 0.84902 1.70451 1.29959e-08 0.197484
tile_temperature / 11256 -407068 1.40123e+12 -8.05307e+08 8.05307e+08 1.61061e+09 -36.1645 2.25407e+08
tile_lw_grey_albedo / 11236 5.60377e-05 106.052 -0.09 0.085 0.175 4.98733e-09 0.0226482
screen_temperature / 11226 110413 2.78958e+12 -1.07374e+09 1.07374e+09 2.14748e+09 9.83549 3.77404e+08
tile_canopy_water / 9365 -407067 1.40123e+12 -8.05307e+08 8.05307e+08 1.61061e+09 -43.4669 2.47121e+08
area_fraction / 26938 0.000525767 1797.66 -1 0.770616 1.77062 1.95177e-08 0.14259
bulk_fraction / 26938 0.000525767 1797.66 -1 0.770616 1.77062 1.95177e-08 0.14259
liquid_fraction / 11801 5.13399e-05 350.829 -0.502809 0.398375 0.901184 4.35047e-09 0.0646092
frozen_fraction / 21595 0.000474351 1458.08 -1 0.770616 1.77062 2.19658e-08 0.152085
ozone / 72663 4.37992e-08 0.0186999 -4.04532e-06 3.2943e-06 7.33962e-06 6.02771e-13 7.1005e-07
lw_up_tile_rts / 11256 0.0784516 967019 -369.663 384.016 753.679 6.96976e-06 128.317
lw_heating_rate_rts / 72704 -4.18205e-07 0.915891 -0.000222405 0.000283027 0.000505432 -5.75217e-12 2.5105e-05
lw_up_tile_rtsi / 11256 0.0238005 342151 -130.928 136.271 267.199 2.11447e-06 45.6822
sw_heating_rate_rtsi / 4370 12.559 4.56587e+07 -1.25056e+06 1.25056e+06 2.50113e+06 0.0028739 107275
lw_heating_rate_rtsi / 72704 12.57 4.56463e+07 -1.25092e+06 1.25092e+06 2.50184e+06 0.000172892 26292.5
tile_snow_mass / 5531 -407097 1.40127e+12 -8.05307e+08 8.05343e+08 1.61065e+09 -73.6029 3.21573e+08
tile_snow_rgrain / 11195 -407067 1.40123e+12 -8.05307e+08 8.05307e+08 1.61061e+09 -36.3615 2.26021e+08
n_snow_layers / 4487 -407067 1.40123e+12 -8.05307e+08 8.05307e+08 1.61061e+09 -90.7215 3.57035e+08
snow_depth / 4487 -407067 1.40123e+12 -8.05307e+08 8.05307e+08 1.61061e+09 -90.7215 3.57035e+08
snowpack_density / 11195 -407067 1.40123e+12 -8.05307e+08 8.05307e+08 1.61061e+09 -36.3615 2.26021e+08
snow_under_canopy / 4487 -407067 1.40123e+12 -8.05307e+08 8.05307e+08 1.61061e+09 -90.7215 3.57035e+08
snow_layer_thickness / 14576 -304125 4.77815e+12 -1.07374e+09 8.05307e+08 1.87905e+09 -20.8648 3.79038e+08
snow_layer_ice_mass / 17515 -304091 4.77816e+12 -1.07378e+09 8.05343e+08 1.87912e+09 -17.3617 3.45777e+08
snow_layer_liq_mass / 14576 -304125 4.77815e+12 -1.07374e+09 8.05307e+08 1.87905e+09 -20.8648 3.79038e+08
snow_layer_temp / 27641 -304125 4.77815e+12 -1.07374e+09 8.05307e+08 1.87905e+09 -11.0027 2.75245e+08
snow_layer_rgrain / 27641 -304125 4.77815e+12 -1.07374e+09 8.05307e+08 1.87905e+09 -11.0027 2.75244e+08
soil_temperature / 4008 -1404.61 3.11385e+11 -1.07374e+09 1.07374e+09 2.14748e+09 -0.350451 2.01508e+08
soil_moisture / 3944 -0.0488899 334394 -894.919 764.066 1658.98 -1.2396e-05 176.294
unfrozen_soil_moisture / 3944 -1.12559e-05 340.285 -0.965407 0.823503 1.78891 -2.85394e-09 0.16016

and for the global model:

Variable Group Count Sum AbsSum Min Max Range Mean StdDev
checkpoint_W3 / 1 -1.10792e+09 1.10792e+09 -1.10792e+09 -1.10792e+09 0 -1.10792e+09 0
Mesh2d / 1 -1.10792e+09 1.10792e+09 -1.10792e+09 -1.10792e+09 0 -1.10792e+09 0
Mesh2d_edge_nodes / 576 0 19244 -1867 1867 3734 0 173.939

[mo-rickywong]

No, that's not it, have you used nccmp? I want to see the differences in the lat lon values in the output. and similarly in the global mesh files you are using as input.

So far I only used ncdiff. Differences with nccmp are, for the regional model:

Variable Group Count Sum AbsSum Min Max Range Mean StdDev checkpoint_Wtheta / 1 -1 1 -1 -1 0 -1 0 checkpoint_W3 / 1 -1.11253e+09 1.11253e+09 -1.11253e+09 -1.11253e+09 0 -1.11253e+09 0 checkpoint_W2 / 1 -1.11253e+09 1.11253e+09 -1.11253e+09 -1.11253e+09 0 -1.11253e+09 0 checkpoint_W2_node_x / 220544 0 145292 -1.98049 2.10612 4.0866 0 0.830915 checkpoint_W2_node_y / 220544 0 103486 -0.705284 0.823914 1.5292 0 0.500886 Mesh2d / 1 -1.11253e+09 1.11253e+09 -1.11253e+09 -1.11253e+09 0 -1.11253e+09 0 Mesh2d_edge_nodes / 192 0 192 -1 1 2 0 1.00261 wvar / 72704 0.00111539 6164.43 -2.45019 2.51847 4.96866 1.53416e-08 0.228017 leaf_area_index / 5062 -inf inf -1.79769e+308 1.79769e+308 inf -inf -nan canopy_height / 5061 -inf inf -1.79769e+308 1.79769e+308 inf -inf -nan tile_fraction / 8897 0.000115625 1078.55 -0.855488 0.84902 1.70451 1.29959e-08 0.197484 tile_temperature / 11256 -407068 1.40123e+12 -8.05307e+08 8.05307e+08 1.61061e+09 -36.1645 2.25407e+08 tile_lw_grey_albedo / 11236 5.60377e-05 106.052 -0.09 0.085 0.175 4.98733e-09 0.0226482 screen_temperature / 11226 110413 2.78958e+12 -1.07374e+09 1.07374e+09 2.14748e+09 9.83549 3.77404e+08 tile_canopy_water / 9365 -407067 1.40123e+12 -8.05307e+08 8.05307e+08 1.61061e+09 -43.4669 2.47121e+08 area_fraction / 26938 0.000525767 1797.66 -1 0.770616 1.77062 1.95177e-08 0.14259 bulk_fraction / 26938 0.000525767 1797.66 -1 0.770616 1.77062 1.95177e-08 0.14259 liquid_fraction / 11801 5.13399e-05 350.829 -0.502809 0.398375 0.901184 4.35047e-09 0.0646092 frozen_fraction / 21595 0.000474351 1458.08 -1 0.770616 1.77062 2.19658e-08 0.152085 ozone / 72663 4.37992e-08 0.0186999 -4.04532e-06 3.2943e-06 7.33962e-06 6.02771e-13 7.1005e-07 lw_up_tile_rts / 11256 0.0784516 967019 -369.663 384.016 753.679 6.96976e-06 128.317 lw_heating_rate_rts / 72704 -4.18205e-07 0.915891 -0.000222405 0.000283027 0.000505432 -5.75217e-12 2.5105e-05 lw_up_tile_rtsi / 11256 0.0238005 342151 -130.928 136.271 267.199 2.11447e-06 45.6822 sw_heating_rate_rtsi / 4370 12.559 4.56587e+07 -1.25056e+06 1.25056e+06 2.50113e+06 0.0028739 107275 lw_heating_rate_rtsi / 72704 12.57 4.56463e+07 -1.25092e+06 1.25092e+06 2.50184e+06 0.000172892 26292.5 tile_snow_mass / 5531 -407097 1.40127e+12 -8.05307e+08 8.05343e+08 1.61065e+09 -73.6029 3.21573e+08 tile_snow_rgrain / 11195 -407067 1.40123e+12 -8.05307e+08 8.05307e+08 1.61061e+09 -36.3615 2.26021e+08 n_snow_layers / 4487 -407067 1.40123e+12 -8.05307e+08 8.05307e+08 1.61061e+09 -90.7215 3.57035e+08 snow_depth / 4487 -407067 1.40123e+12 -8.05307e+08 8.05307e+08 1.61061e+09 -90.7215 3.57035e+08 snowpack_density / 11195 -407067 1.40123e+12 -8.05307e+08 8.05307e+08 1.61061e+09 -36.3615 2.26021e+08 snow_under_canopy / 4487 -407067 1.40123e+12 -8.05307e+08 8.05307e+08 1.61061e+09 -90.7215 3.57035e+08 snow_layer_thickness / 14576 -304125 4.77815e+12 -1.07374e+09 8.05307e+08 1.87905e+09 -20.8648 3.79038e+08 snow_layer_ice_mass / 17515 -304091 4.77816e+12 -1.07378e+09 8.05343e+08 1.87912e+09 -17.3617 3.45777e+08 snow_layer_liq_mass / 14576 -304125 4.77815e+12 -1.07374e+09 8.05307e+08 1.87905e+09 -20.8648 3.79038e+08 snow_layer_temp / 27641 -304125 4.77815e+12 -1.07374e+09 8.05307e+08 1.87905e+09 -11.0027 2.75245e+08 snow_layer_rgrain / 27641 -304125 4.77815e+12 -1.07374e+09 8.05307e+08 1.87905e+09 -11.0027 2.75244e+08 soil_temperature / 4008 -1404.61 3.11385e+11 -1.07374e+09 1.07374e+09 2.14748e+09 -0.350451 2.01508e+08 soil_moisture / 3944 -0.0488899 334394 -894.919 764.066 1658.98 -1.2396e-05 176.294 unfrozen_soil_moisture / 3944 -1.12559e-05 340.285 -0.965407 0.823503 1.78891 -2.85394e-09 0.16016

and for the global model:

Variable Group Count Sum AbsSum Min Max Range Mean StdDev checkpoint_W3 / 1 -1.10792e+09 1.10792e+09 -1.10792e+09 -1.10792e+09 0 -1.10792e+09 0 Mesh2d / 1 -1.10792e+09 1.10792e+09 -1.10792e+09 -1.10792e+09 0 -1.10792e+09 0 Mesh2d_edge_nodes / 576 0 19244 -1867 1867 3734 0 173.939

Sorry Juan,
The posted output just isn't human-friendly. Please post the location out your output files from the 1cpu and multiple cpu runs so I can have a look at them.

[ukmo-juan-castillo]

The output for 1 and multiple CPUs are on VDI:

global to global

1 CPU
/home/users/juan.m.castillo/cylc-run/lfric2lfric_main/run1/work/1/run_lfric2lfric_clim_gal9-C24_C12_azspice_gnu_fast-debug-64bit/lfric_checkpoint_write.nc

6 CPUs
/home/users/juan.m.castillo/cylc-run/lfric2lfric_multi_CPU/run1/work/1/run_lfric2lfric_clim_gal9-C24_C12_azspice_gnu_fast-debug-64bit/lfric_checkpoint_write.nc

regional to regional

1 CPU
/home/users/juan.m.castillo/cylc-run/lfric2lfric_main/run1/work/1/run_lfric2lfric_ral_seuk-C32_lam_MG_azspice_gnu_fast-debug-64bit/lfric_checkpoint_write.nc

4 CPUs
/home/users/juan.m.castillo/cylc-run/lfric2lfric_multi_CPU/run1/work/1/run_lfric2lfric_ral_seuk-C32_lam_MG_azspice_gnu_fast-debug-64bit/lfric_checkpoint_write.nc

[mo-rickywong]

Global2Global differences between 1xCPU and 6xCPU

• The edge node differences here are the same connectivity, though the orientation is transposed. This has been seen before as coming up with differences, although nothing as far as I know uses this connectivity (though I could be wrong).

• Other differences appear to be with the id number of the field (I think), though I don't know how these are determined by the code that writes to file.

The Regional to Regional differences between 1xCPU and 4xCPU

• Fields don't match, I think you posted the wrong files.

Note you posted output for run_lfric2lfric_ral_seuk-C32_lam_MG_azspice_gnu_fast-debug-64bit. Though as we know the presence of intergrid maps will trigger checks that cause this to fail. So what if anything was done to get this to run? if this has been modified, I'd need to know.

[ukmo-juan-castillo]

Global2Global differences between 1xCPU and 6xCPU

* The edge node differences here are the same connectivity, though the orientation is transposed. This has been seen before as coming up with differences, although nothing as far as I know uses this connectivity (though I could be wrong).

* Other differences appear to be with the id number of the field (I think), though I don't know how these are determined by the code that writes to file.

### The Regional to Regional differences between 1xCPU and 4xCPU

* Fields don't match, I think you posted the wrong files.

Note you posted output for run_lfric2lfric_ral_seuk-C32_lam_MG_azspice_gnu_fast-debug-64bit. Though as we know the presence of intergrid maps will trigger checks that cause this to fail. So what if anything was done to get this to run? if this has been modified, I'd need to know.

I have updated the paths to the output files, sorry for the mistake.

I have not changed anything to be able to run using the multi grid. The only thing that is needed, I believe, is using a number of CPUs that divides the number of grid cells in each of the grids. I will open a new issue to deal with multigrids.

[mo-rickywong]

Global2Global differences between 1xCPU and 6xCPU

* The edge node differences here are the same connectivity, though the orientation is transposed. This has been seen before as coming up with differences, although nothing as far as I know uses this connectivity (though I could be wrong).

``` * Other differences appear to be with the id number of the field (I think), though I don't know how these are determined by the code that writes to file. ``` ### The Regional to Regional differences between 1xCPU and 4xCPU ``` * Fields don't match, I think you posted the wrong files. ```

Note you posted output for run_lfric2lfric_ral_seuk-C32_lam_MG_azspice_gnu_fast-debug-64bit. Though as we know the presence of intergrid maps will trigger checks that cause this to fail. So what if anything was done to get this to run? if this has been modified, I'd need to know.

I have updated the paths to the output files, sorry for the mistake.

I have not changed anything to be able to run using the multi grid. The only thing that is needed, I believe, is using a number of CPUs that divides the number of grid cells in each of the grids. I will open a new issue to deal with multigrids.

Your branch has set ral_seuk-C32_lam_MG to run with 4 cpus and changes the KGOS, but you have not posted the output. That is what I'm referring to in the previous message.

Thank you for the updated post, though there is still the missing output for the oasisi_clim_gal9-C24_C12 runs

[mo-rickywong]

Glad that you appear to have found a fix for the issue. Although there are 2 issues here:

1. The rose metadata implementation, which allows the intended namelists to reach each of the CPUs and allow the program to run.
2. The iodef.xml definition, which corrects the output that lfric2lfric produces, when run with multiple CPUs.

These are 2 different issues. This ticket is for the 1st issue. The fix to the output is technically a separate issue.

If you want to progress this PR you may do so, but not change the rose-suite. If you want to add a multi-cpu test to the test-suite on this PR, then it can't go on without a fix as well.

I'd suggest keeping the issues separate.

• This PR for the metadata.
• A follow up PR to add a multi-cpu run with the fix that corrects the output when run with multiple cpus.

It keeps the scope of the PRs clear.

[ukmo-juan-castillo]

Glad that you appear to have found a fix for the issue. Although there are 2 issues here:

1. The rose metadata implementation, which allows the intended namelists to reach each of the CPUs and allow the program to run.

2. The iodef.xml definition, which corrects the output that lfric2lfric produces, when run with multiple CPUs.

These are 2 different issues. This ticket is for the 1st issue. The fix to the output is technically a separate issue.

If you want to progress this PR you may do so, but not change the rose-suite. If you want to add a multi-cpu test to the test-suite on this PR, then it can't go on without a fix as well.

I'd suggest keeping the issues separate.

* This PR for the metadata.

* A follow up PR to add a multi-cpu run with the fix that corrects the output when run with multiple cpus.

It keeps the scope of the PRs clear.

Thanks for your comments. I would prefer having all changes in the same PR though for mainly two reasons:

1. If we just fix lfric2lfric so that it can run using multiple CPUS without adding multiple CPU tests, then we will not be testing the new functionality we are introducing
2. Time: we have a deadline at the end of this month to have this functionality in the trunk, so having multiple PRs will slow things down

Furthermore, the fix does not imply any changes to xml files, but to kernels.

I will add all changes (they are not so many) and send test, and when you have a look at the final product you still consider we need two PRs, then I can easily split it.

[ukmo-juan-castillo]

Following the reviewer's suggestions, this PR will now only consist in adding the capability of running in multiple CPUs.

The trac.log has been updated. Differences of the test branch with the original branch are due only to the removal of old KGO files (as some of the KGO tests changed name). There are some tests that do not pass, but I guess they are present in the trunk as the changes in this PR only refer to lfric2lfric.

[mo-rickywong]

Following the reviewer's suggestions, this PR will now only consist in adding the capability of running in multiple CPUs.

The trac.log has been updated. Differences of the test branch with the original branch are due only to the removal of old KGO files (as some of the KGO tests changed name). There are some tests that do not pass, but I guess they are present in the trunk as the changes in this PR only refer to lfric2lfric.

@ukmo-juan-castillo You have re-requested a review with the view to only applying the changes that allow lfric2lfric to run with multiple cpus. However, the branch you have asked to be re-viewed on the PR has not changed since I last looked at it a month ago, have you pushed your changes to the branch associated with the PR?

[ukmo-juan-castillo]

I updated as suggested by the reviewer, run again all rose stem tests, and updated trac.log in the PR description.

[mo-rickywong]

I've run this fric2lfric branch on a local copy merged up to head of main and existing results don't change. I also modified the lfric2lfric tasks n the test-suite to use 6 mpi ranks to check the failure modes. The tasks which fail to run all complain within partitioning defensive checks as expected.

As a note:
This PR contains an upgrade macro and has had the rose stem app update. As per the SSD process guide, the rose-configurations shouldn't be updated directly (https://metoffice.github.io/simulation-systems/Development/inputs.html). That said I don't think it's an issue here really as the change is limited to lfric2lfric and is not extensive.

[mo-rickywong]

SciTech review now fine.