diff --git a/docs/models/run-a-model/run-access-esm.md b/docs/models/run-a-model/run-access-esm.md index d012ce066..a0dd8463d 100644 --- a/docs/models/run-a-model/run-access-esm.md +++ b/docs/models/run-a-model/run-access-esm.md @@ -63,7 +63,7 @@ For the general requirements needed to run all ACCESS models, please refer to th A suitable ACCESS-ESM pre-industrial configuration is avaible on the coecms GitHub.
In order to get it, on Gadi, create a directory where to keep the model configuration, and clone the GitHub repo in it by running: -
git clone https://github.com/coecms/esm-pre-industrial
+
git clone https://github.com/coecms/esm-pre-industrial.git
mkdir -p ~/access-esm cd ~/access-esm diff --git a/docs/models/run-a-model/run-access-om.md b/docs/models/run-a-model/run-access-om.md index df2873c0c..12f2ef974 100644 --- a/docs/models/run-a-model/run-access-om.md +++ b/docs/models/run-a-model/run-access-om.md @@ -1,4 +1,396 @@ -# Running ACCESS-OM2 Model + +{% set model = "ACCESS-OM" %} +# Run {{ model }} +## Requirements +
+Before running {{ model }}, you need to make sure to possess the right tools and to have an account with specific institutions. +
+ +### General requirements +
+For the general requirements needed to run all ACCESS models, please refer to the Getting Started (TO DO check link) page. +
+ +### Model-specific requirements +
+ +
+---------------------------------------------------------------------------------------- + +## Get {{ model }} configuration +
+A standard {{ model }} configuration is avaible on the COSIMA GitHub. +
+This is a 1° horizontal resolution configuration, with interannual forcing from 1 Jan 1958 to 31 Dec 2018. +
+In order to get it, on gadi, create a directory where to keep the model configuration, and clone the GitHub repo in it by running: +
git clone https://github.com/COSIMA/1deg_jra55_iaf.git
+ + mkdir -p ~/access-om + cd ~/access-eom + git clone https://github.com/COSIMA/1deg_jra55_iaf.git + Cloning into '1deg_jra55_iaf'... + remote: Enumerating objects: 14715, done. + remote: Counting objects: 100% (3401/3401), done. + remote: Compressing objects: 100% (24/24), done. + remote: Total 14715 (delta 3383), reused 3379 (delta 3377), pack-reused 11314 + Receiving objects: 100% (14715/14715), 35.68 MiB | 18.11 MiB/s, done. + Resolving deltas: 100% (10707/10707), done. + +Note: Some modules might interfere with the git commands (for example matlab/R2018a). If you are running into issues during the cloning of the repository, it might be a good idea to run 
module purge
first, before trying again. +
+---------------------------------------------------------------------------------------- + +## Edit {{ model }} configuration +
+In order to modify an {{ model }} configuration, it is worth understanding a bit more how its job scheduler payu works. +
+ +### Payu +
+Payu is a workflow management tool for running numerical models in supercomputing environments. +
+The general layout of a payu-supported model run consists of two main directories: + +This distinction of directories keeps the small-size configuration files separated from the larger binary outputs and inputs. In this way, we can place the configuration files in the $HOME directory (being the only filesystem on gadi that is actively backed up), without overloading it with too much data. +
+Moreover, this separation allows to run multiple self-resubmitting experiments simultaneously that might share common executables and input data. +
+To proceed with the setup of the laboratory directory, from the control directory run: +
payu init
+This will create the laboratory directory, along with other subdirectories (depending on the configuration). The main subdirectories we are interested in are: + +
+ +### Edit the Master Configuration file +
+The config.yaml file, located in the control directory, is the Master Configuration file. +
+This file controls the general model configuration and if we open it in a text editor, we can see different parts: + +To know more about other configuration settings for the config.yaml file, please check how to configure your experiment with payu. +
+---------------------------------------------------------------------------------------- + +## Run {{ model }} configuration +After editing the configuration, we are ready to run {{ model }}. +
+{{ model }} suites run on gadi through a PBS job submission managed by payu. + +### Payu setup (optional) +
+As a first step, from the control directory, is good practice to run: +
payu setup
+This will prepare the model run, based on the experiment configuration. + + payu setup + laboratory path: /scratch/$PROJECT/$USER/access-esm + binary path: /scratch/$PROJECT/$USER/access-esm/bin + input path: /scratch/$PROJECT/$USER/access-esm/input + work path: /scratch/$PROJECT/$USER/access-esm/work + archive path: /scratch/$PROJECT/$USER/access-esm/archive + Loading input manifest: manifests/input.yaml + Loading restart manifest: manifests/restart.yaml + Loading exe manifest: manifests/exe.yaml + Setting up atmosphere + Setting up ocean + Setting up ice + Setting up coupler + Checking exe and input manifests + Updating full hashes for 3 files in manifests/exe.yaml + Creating restart manifest + Updating full hashes for 30 files in manifests/restart.yaml + Writing manifests/restart.yaml + Writing manifests/exe.yaml + +Note: You can skip this step as it is included also in the run command. However, runnning it explicitly helps to check for errors and make sure executable and restart directories are accessible. +
+ +### Run configuration +
+To run {{ model }} configuration for one internal run length (controlled by runtime in the config.yaml file), run: +
payu run -f
+This will submit a single job to the queue with a total run length of runtime. It there is no previous run, it will start from the start date indicated in the config.yaml file, otherwise it will perform a warm restart from a precedently saved restart file. +
+Note:The -f option ensures that payu will run even if there is an existing non-empty work directory, which happens if a run crashes. + + payu run -f + Loading input manifest: manifests/input.yaml + Loading restart manifest: manifests/restart.yaml + Loading exe manifest: manifests/exe.yaml + payu: Found modules in /opt/Modules/v4.3.0 + qsub -q normal -P <project> -l walltime=11400 -l ncpus=384 -l mem=1536GB -N pre-industrial -l wd -j n -v PAYU_PATH=/g/data/hh5/public/apps/miniconda3/envs/analysis3-23.01/bin,MODULESHOME=/opt/Modules/v4.3.0,MODULES_CMD=/opt/Modules/v4.3.0/libexec/modulecmd.tcl,MODULEPATH=/g/data3/hh5/public/modules:/etc/scl/modulefiles:/opt/Modules/modulefiles:/opt/Modules/v4.3.0/modulefiles:/apps/Modules/modulefiles -W umask=027 -l storage=gdata/access+gdata/hh5 -- /g/data/hh5/public/apps/miniconda3/envs/analysis3-23.01/bin/python3.9 /g/data/hh5/public/apps/miniconda3/envs/analysis3-23.01/bin/payu-run + <job-ID>.gadi-pbs + +
+ +### Run configuration for multiple years +
+If you want to run {{ model }} configuration for multiple internal run lengths (controlled by runtime in the config.yaml file), you can use the option -n: +
payu run -n <number-of-runs>
+This will run the configuration number-of-runs times with a total run length of runtime * number-of-runs. The number of consecutive PBS jobs submitted to the queue depends on the runspersub value specified in the config.yaml file. +
+ +### Understand runtime, runspersub, and -n parameters +
+With the correct use of runtime, runspersub, and -n parameters, we can have full control of our run. +
+ +Let's have some practical examples: + +
+---------------------------------------------------------------------------------------- + +## Monitor {{ model }} runs +
+Currently, there is no specific tool to monitor {{ model }} runs. +
+One way to check the status of our run is running: +
qstat -u $USER
+This will show the status of all your PBS jobs (if there is any PBS job submitted): + + qstat -u $USER + Job id                Name             User             Time Use S Queue + --------------------- ---------------- ---------------- -------- - ----- + <job-ID>.gadi-pbs     pre-industrial   <$USER>            <time> R normal-exec + <job-ID>.gadi-pbs     <other-job-name> <$USER>            <time> R normal-exec + <job-ID>.gadi-pbs     <other-job-name> <$USER>            <time> R normal-exec + +If you changed the jobname in the PBS resources of the Master Configuration file, that will be your job's Name instead of pre-industrial. +
+S indicates the status of your run: + +If there is no listed job with your jobname (or if there is no job submitted at all), your run might have successfully completed, or might have been terminated due to an error. +
+ +### Error and output log files +
+While the model is running, payu saves the standard output and standard error into the files access.out and access.err in the control directory. You can examine these files, as the run progresses, to check on it's status. +
+After the model has completed its run, or if it crashed, the output and error log files, respectively, are renamed by default into jobname.o<job-ID> and jobname.e<job-ID>. +
+---------------------------------------------------------------------------------------- + +## {{ model }} outputs +
+While the configuration is running, output files (as well as restart files) are moved from the work directory to the archive directory, under /scratch/$PROJECT/$USER/access-esm/archive (also symlinked in the control directory under ~/access-esm/esm-pre-industrial/archive). +
+Both outputs and restarts are stored into subfolders for each different configuration (esm-pre-industrial in our case), and inside the configuration folder, they are subdivided for each internal run. +
+The format of a typical output folder is outputXXX, whereas the typical restart folder is usually formatted as restartXXX, with XXX being the number of internal run, starting from 000. +
+In the respective folders, outputs and restarts are separated for each model component. +
+For the atmospheric output data, each file it is usually a UM fieldsfile, formatted as <UM-suite-identifier>a.p<output-stream-identifier><time-identifier>. + + cd /scratch/$PROJECT/$USER/access-esm/archive/esm-pre-industrial + ls + output000 pbs_logs restart000 + ls output000/atmosphere + aiihca.daa1210 aiihca.daa1810 aiihca.paa1apr aiihca.paa1jun aiihca.pea1apr aiihca.pea1jun aiihca.pga1apr aiihca.pga1jun atm.fort6.pe0 exstat ihist prefix.CNTLGEN UAFLDS_A aiihca.daa1310 aiihca.daa1910 aiihca.paa1aug aiihca.paa1mar aiihca.pea1aug aiihca.pea1mar aiihca.pga1aug aiihca.pga1mar cable.nml fort.57 INITHIS prefix.PRESM_A um_env.py aiihca.daa1410 aiihca.daa1a10 aiihca.paa1dec aiihca.paa1may aiihca.pea1dec aiihca.pea1may aiihca.pga1dec aiihca.pga1may CNTLALL ftxx input_atm.nml SIZES xhist aiihca.daa1510 aiihca.daa1b10 aiihca.paa1feb aiihca.paa1nov aiihca.pea1feb aiihca.pea1nov aiihca.pga1feb aiihca.pga1nov CONTCNTL ftxx.new namelists STASHC aiihca.daa1610 aiihca.daa1c10 aiihca.paa1jan aiihca.paa1oct aiihca.pea1jan aiihca.pea1oct aiihca.pga1jan aiihca.pga1oct debug.root.01 ftxx.vars nout.000000 thist aiihca.daa1710 aiihca.daa2110 aiihca.paa1jul aiihca.paa1sep aiihca.pea1jul aiihca.pea1sep aiihca.pga1jul aiihca.pga1sep errflag hnlist prefix.CNTLATM UAFILES_A + + +
+ + + +
+
References
+ + Welcome to **ACCESS-OM2** — a coupled ocean-ice model and collection of [configurations](http://cosima.org.au/index.php/models/) developed by the [Consortium for Ocean-Sea Ice Modelling in Australia (COSIMA)](http://cosima.org.au/). @@ -12,6 +404,8 @@ The configurations available here are updated from the version 1.0 configuration Non-NCI users can access a subset of the ACCESS-OM2 output via the [COSIMA Model Output Collection](https://dx.doi.org/10.4225/41/5a2dc8543105a). + + ## How to run ACCESS-OM2 Start by reading the [[Quick start\|Getting-started#quick-start]] guide. If you are using [gadi.nci.org.au](http://nci.org.au/our-systems/hpc-systems) at the [NCI National Facility](http://nci.org.au/) and are happy to use our pre-compiled executables then this should be all you need. The page also provides instructions for building your own executables.