Are there any linked Issues or Pull Requests?
No response
Brief description
Currently, the lfric_atm rose-stem suite only checks the "known good output" (KGO) pertinent to model evolution, defined by global checksums on the 5 main prognostic variables in the model (wind, temperature, pressure, density, moist mass mixing ratios).
This is not sufficient testing to ensure that the behaviour of the modelling system is preserved. None of these 5 variables are used for driving the data assimilation system, downstream products or evaluating the skill of the model - these processes are entirely reliant on diagnostic output constructed within the model. Therefore, rose-stem needs to check that this diagnostic output is preserved. It would be entirely possible for the current KGO checks to pass but diagnostic output to be altered in a way that could significantly (negatively) impact our ability to run and use the model.
Therefore rose-stem checks need adding for diagnostic output files. Usually (e.g. in the UM and standalone LFRic suites) this process is done with "rose-ana" tasks, an example of which can be seen here: https://code.metoffice.gov.uk/trac/roses-u/browser/d/i/6/2/0/trunk/app/rose_ana_restart/rose-app.conf
This uses "nccmp" to compare LFRic output files. Ideally the exclude list should be empty, although it is possible that some diagnostics do not currently contain reproducible missing data in areas where the diagnostic is invalid (e.g. land temperature over the sea). If during adding the tests, we find diagnostics for which this is the case, they should be added to the exclude list and separate PRs should populate the missing data correctly and remove them from the exclude list.
The output files which should be checked are those files produced by the following xml files:
Further details of the issue.
No response
Are there any linked Issues or Pull Requests?
No response
Brief description
Currently, the lfric_atm rose-stem suite only checks the "known good output" (KGO) pertinent to model evolution, defined by global checksums on the 5 main prognostic variables in the model (wind, temperature, pressure, density, moist mass mixing ratios).
This is not sufficient testing to ensure that the behaviour of the modelling system is preserved. None of these 5 variables are used for driving the data assimilation system, downstream products or evaluating the skill of the model - these processes are entirely reliant on diagnostic output constructed within the model. Therefore, rose-stem needs to check that this diagnostic output is preserved. It would be entirely possible for the current KGO checks to pass but diagnostic output to be altered in a way that could significantly (negatively) impact our ability to run and use the model.
Therefore rose-stem checks need adding for diagnostic output files. Usually (e.g. in the UM and standalone LFRic suites) this process is done with "rose-ana" tasks, an example of which can be seen here: https://code.metoffice.gov.uk/trac/roses-u/browser/d/i/6/2/0/trunk/app/rose_ana_restart/rose-app.conf
This uses "nccmp" to compare LFRic output files. Ideally the exclude list should be empty, although it is possible that some diagnostics do not currently contain reproducible missing data in areas where the diagnostic is invalid (e.g. land temperature over the sea). If during adding the tests, we find diagnostics for which this is the case, they should be added to the exclude list and separate PRs should populate the missing data correctly and remove them from the exclude list.
The output files which should be checked are those files produced by the following xml files:
Further details of the issue.
No response