Mpas testing #2
Open
Mpas testing #2
+1,801
−5
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
some changes to tests config some scripts to support performance benchmarking adding nvhpc-cpu some changes to exit codes
Adding reference generation capability and a separate jsonc
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR builds on MPAS-Dev#1284 to add runtime tests for MPAS.
To launch tests from the root directory
The following toolchain/builds are proposed
gnu,gnu-double,gpu-openmp,gnu-debugnvhpc-cpu,nvhpc-gpu,nvhpc-gpu-doubleintel,intel-doubleThe following test scenarios are proposed:
Jablonowski-Williamson baroclinic wave (dycore)Limited area over CONUSAquaplanetSquall lineThe following tests are proposed, orthogonal to test scenarios above.
Base: The simplest test configuration for a given scenario. The output restart file is checked for bit identical results against a stored reference.Restart: Tests the ability to proceed from an intermediate restart file (provided by base run) and produce results bit-identical to base run.Parallel: Tests for various parallel implementations/configurations. The resulting restart files are checked for bit-identical results against the base runs. This is further classified intoMPIOMPMultinode CPUMulti GPUMultinode GPUPerformance: Five instances of a shorter run are performed and the timing statistics are compared against references stored in a database. The idea would be to run a nightly build + performance runs for the develop branch, and store the statistics in the SQLite databse. When the performance check is run for PRs, then we could use the average of the last 5-10 days to decide if the performance of our PR has regressed.Test matrix:
The proposal is to avoid running all possible combinations of (toolchain X scenario X tests) and instead carefully map out a minimum set of tests in this space that would provide a reasonable guarantee of correctness and keep a eye on performance regression.
GNU (OPTIMIZED)GNU (DEBUG)- Build onlyGNU - DOUBLE PRECISION (OPTIMIZED)NVHPC-CPU (OPTIMIZED)NVHPC-GPU (OPTIMIZED)NVHPC-GPU DOUBLE (OPTIMIZED)INTEL (OPTIMIZED)INTEL (DEBUG)- Build onlyReference generation
To periodically update the reference restart files against which the base runs are compare, we could use another json file
.ci/mpas_refgen.jsoncto generate these references from the base branch (develop). This would be triggered manually as required, when either the base branch introduces substantial changes or the GPU porting of a routine results in non BFB identical results (which have been otherwise verified).To launch the reference generation from the root directory