Initial testing framework simplified #1284
Conversation
In order to run test scripts outside of a testing framework, handling of the environment setup should not be solely dependent on running within a dedicated test framework. This has the added benefit of compartmentalizing the duties of environment and dependency solving from running the tests. These environment scripts allow for the selection of a particular environment with the default being the fqdn of the current host. From there, arguments are routed using standard POSIX-sh to a respective script. In the case of Derecho (applicable to any system using lmod) all subsequent argument are treated as modules to load into the current session. The hostenv.sh script relies on one "argument" $AS_HOST being passed in via variable setting to facilitate selection. The helpers.sh script provides convenience features for substring checking in sh, delayed environment variable expansion via eval, and quick banner creation. The derecho.sh script is included as the first supported environment.
This script will facilitate the first tests. There are only three requirements of any given test script with the planned testing framework If a different testing framework is used in the future, these requirements of the test scripts can and should be re-evaluated. The test script should : 1. Take the intended host / configuration env as the first argument 2. Take the working directory which we cd to as the second argument 3. Output some key phrase at the end of the test to denote success, anything else (non-zero exit code, no phrase but return zero) is considered failure This particular compilation test script satisfies the above while also providing enough flexibility to select core, target, parallel jobs, and other command-line options into the make build. Additionally, for convenience environment variables can be passed in as command-line options to the test script to modularize certain inputs.
Following the documentation of the hpc-workflows testing framework and the testing structure found in .ci/, a JSON file for a GNU compilation test was added. This test will compile the atmosphere core using gnu + single precision. If this test is run using the derecho configuration the appropriate modules will attempt to be loaded. For non-derecho environments, per the testing structure under .ci/, if no configuration exists in .ci/hostenv.sh then the current environment will be used verbatim.
|
|
||
| echo "Compiling with options $target core=$core $debug $buildCommand" | ||
| echo "make $target CORE=$core $debug $buildCommand" | ||
| make $target CORE=$core $debug $buildCommand |
There was a problem hiding this comment.
Was wondering if there's value to including the other MPAS build options in this PR, or if it's better in a succeeding PR.
| // We want NUM_PROCS to match ncpus | ||
| "base_env_numprocs" : [ "-e", "NUM_PROCS=16" ], | ||
| "very_last_modules" : [ "cray-mpich", "parallel-netcdf" ], | ||
| ".*gnu.*::test_modules" : [ "gcc/12.2.0" ] |
There was a problem hiding this comment.
One part where I've had some troubles with is the ordering of the arguments that are passed to the script we are running (build.sh here). If I'm not careful enough, there are sometimes options I'm trying to pass to the script (say "-z option1") that get misconstrued as modules to be loaded, etc.
Probably related to the ordering of options which are defined in these regex statements (".make.::args_build_options"). I usually avoid this issue by specifying the arguments locally after every command ("command" : ".ci/tests/build.sh"). Not sure if I what I wrote is very clear, but I'll try to clarify my comment with a better example.
There was a problem hiding this comment.
I think I understand. This is probably due to the way I have it setup where on the first unknown arg it encounters getopts exits and forwards the rest to the environment script.
The order passed in will be determined by the alphabetical order of the non-regex portions, deconflicted by order of appearance in the config file. I'm not sure if you've already found this documentation (https://github.com/islas/hpc-workflows/blob/main/tutorials/AdvancedTestConfig_regex_argpacks.md#simple-regex-argpack-config) at the end of that section there is a note about this.
There was a problem hiding this comment.
It is, admittedly, a very fragile setup relying on argument order between what the test script needs and what the environment script needs..
Perhaps a better approach would be something like the -e env option where things are accumulated and then handled afterwards. In that way multiple things can be passed to then env script without relying on order
There was a problem hiding this comment.
Thanks @islas! I don't think I've looked at this specific section of your documentation. I will go through that.
I think it's a relatively minor issue, and as long as there's some documentation, I'd be fine with things as they are.
This PR introduces testing capabilities through a series of compartmentalized commits :
The tests can be run using the following command :
.ci/hpc-workflows/.ci/runner.py .ci/mpas_compilation.jsonc -t gnuIf this is run in an environment noted to have PBS/SLURM (as seen by the line in
mpas_compilation.jsonc:"hsn.de.hpc" : { "submission" : "PBS" }) an account key must be provided to utilize job submission. Local testing in within the login node can be forced but should be used with caution.This PR is meant to supersede PR #1218 as a smaller incremental change.