From 71f7bae47608c634bee8db84b4fc88fe70528711 Mon Sep 17 00:00:00 2001 From: Davide Marchegiani Date: Fri, 12 Jul 2024 15:22:30 +1000 Subject: [PATCH 01/14] Copied how to run ACCESS-OM page into how to run ACCESS-ESM and applied some changes. --- docs/models/run-a-model/run-access-esm.md | 802 ++++++++++-------- .../models/run-a-model/run-access-esm__old.md | 470 ++++++++++ docs/models/run-a-model/run-access-om.md | 53 +- 3 files changed, 950 insertions(+), 375 deletions(-) create mode 100644 docs/models/run-a-model/run-access-esm__old.md diff --git a/docs/models/run-a-model/run-access-esm.md b/docs/models/run-a-model/run-access-esm.md index 6b4c74e89..a7744d5f9 100644 --- a/docs/models/run-a-model/run-access-esm.md +++ b/docs/models/run-a-model/run-access-esm.md @@ -1,470 +1,574 @@ -{% set model = "ACCESS-ESM" %} +{% set model = "ACCESS-ESM1.5" %} +{% set github_configs = "https://github.com/ACCESS-NRI/access-esm1.5-configs" %} [PBS job]: https://opus.nci.org.au/display/Help/4.+PBS+Jobs +[gadi]: https://opus.nci.org.au/display/Help/0.+Welcome+to+Gadi#id-0.WelcometoGadi-Overview +[payu]: https://github.com/payu-org/payu +[model components]: /models/configurations/access-esm/#model-components +[model configurations]: /models/configurations/access-esm/#access-esm15 -!!! bug - There is a known issue with the {{ model }} configurations currently set up in the GitHub repository below. - {{ model }} configurations are currently being updated and an ACCESS-NRI release will come - in the following weeks. - If you encounter problems with the current guide in this page, consider asking for help on the [ACCESS-Hive Forum](https://forum.access-hive.org.au). +[:fontawesome-brands-github:{: class="twemoji icon-before-text"} {{ model }} configurations]({{github_configs}}){: class="text-card"} # Run {{ model }} +## About + +{{ model }} is a fully-coupled global climate model. More information is available in the [{{ model }} overview][model configurations]. + +The instructions below outline how to run {{ model }} using ACCESS-NRI's software deployment pipeline, specifically designed to run on the [National Computating Infrastructure (NCI)](https://nci.org.au/about-us/who-we-are) supercomputer [_Gadi_][gadi]. + +If you are unsure whether {{ model }} is the right choice for your experiment, take a look at the overview of [ACCESS Models](/models). + +All {{model}} configurations are open source, licensed under [CC BY 4.0](https://creativecommons.org/licenses/by/4.0/?ref=chooser-v1")![CC icon](https://mirrors.creativecommons.org/presskit/icons/cc.svg?ref=chooser-v1){: style="height:1em;margin-left:0.2em;vertical-align:text-top;"}![BY icon](https://mirrors.creativecommons.org/presskit/icons/by.svg?ref=chooser-v1){: style="height:1em;margin-left:0.2em;vertical-align:text-top;"} and available on [ACCESS-NRI GitHub]({{github_configs}}). + +{{ model }} release notes are [available on the ACCESS-Hive Forum **TODO change with correct link**](https://forum.access-hive.org.au/t/access-om2-release-information/1602/) and are updated when new releases are made available. + ## Prerequisites ### General prerequisites Before running {{ model }}, you need to fulfil general prerequisites outlined in the [First Steps](/getting_started/first_steps) section. -If you are unsure whether {{ model }} is the right choice for your experiment, take a look at the overview of [ACCESS Models](/models). - ### Model-specific prerequisites -{{ model }} is installed on NCI's supercomputer _Gadi_ and uses [_payu_](#payu), a tool for running and managing model experiments. Following these prerequisites ensures you have access to this infrastructure. - -- **Join the _access_ and _hh5_ projects at NCI**
- To join these projects, request membership on the respective [access](https://my.nci.org.au/mancini/project/access/join) and [hh5](https://my.nci.org.au/mancini/project/hh5/join) NCI project pages.
+- **Join the _vk83_ project at NCI**
+ To join this project, request membership on the [vk83](https://my.nci.org.au/mancini/project/vk83/join) NCI project page.
For more information on joining specific NCI projects, refer to [How to connect to a project](https://opus.nci.org.au/display/Help/How+to+connect+to+a+project). + - **Payu**
- _Payu_ on _Gadi_ is available through the `conda/analysis3` environment in the _hh5_ project.
- After obtaining _hh5_ project membership, load the `conda/analysis3` environment to automatically retrieve _payu_ as follows: - ``` - module use /g/data/hh5/public/modules - module load conda/analysis3 - ``` + [Payu][payu] is a workflow management tool for running numerical models in supercomputing environments, for which there is extensive [documentation](https://payu.readthedocs.io/en/latest/).
+ _Payu_ on _Gadi_ is available through a dedicated `conda` environment in the _vk83_ project.
+ After joining the _vk83_ project, load the `payu` module: + + module use /g/data/vk83/modules + module load payu + To check that _payu_ is available, run: - ``` - payu --version - ``` + + payu --version + payu --version - 1.0.19 + 1.1.3 + !!! warning + _payu_ version >=1.1.3 is required + TODO: Check if any specific version is required + +---------------------------------------------------------------------------------------- ## Get {{ model }} configuration -{{ model }} configurations are available in the [coecms GitHub](https://github.com/coecms/access-esm) repository.
-To get it on _Gadi_, clone the {{ model }} GitHub repository by running: -``` -git clone https://github.com/coecms/access-esm.git -``` -This will create the `access-esm` folder. - - git clone https://github.com/coecms/access-esm.git - Cloning into 'access-esm'... - remote: Enumerating objects: 1625, done. - remote: Counting objects: 100% (1625/1625), done. - remote: Compressing objects: 100% (575/575), done. - remote: Total 1625 (delta 1042), reused 1621 (delta 1040), pack-reused 0 - Receiving objects: 100% (1625/1625), 2.79 MiB | 11.24 MiB/s, done. - Resolving deltas: 100% (1042/1042), done. - ls ~/access-esm - atmosphere config.yaml coupler ice manifests ocean README.md - +All released {{ model }} configurations are available from the [{{ model }} configs]({{ github_configs }}) GitHub repository.
+Released configurations are tested and supported by ACCESS-NRI, as an adaptation of those originally developed by [CLEX CMS](https://github.com/coecms/access-esm). -!!! warning - Some modules may interfere with `git` commands (e.g., matlab/R2018a). If you have trouble cloning the repository, run the following command before trying again: - ``` - module purge - ``` - After this step, don't forget to reload the `conda/analysis3` module to retrieve _payu_, as specified in the [Model-specific prerequisites](#model-specific-prerequisites) section. +For more information on {{ model }} configurations, check [{{model}}][model configurations] page. -Each {{ model }} configuration is stored as a specially-named branch in the {{ model }} GitHub repository. -To check all the available branches on the repo, run the following command inside the newly-created `access-esm` folder: -``` -git branch -a -``` - - cd ~/access-esm - git branch -a - ٭ main -   remotes/origin/82ka -   remotes/origin/HEAD → origin/main -   remotes/origin/ccarouge-patch-1 -   remotes/origin/historical -   remotes/origin/last-interglacial -   remotes/origin/last-millenium -   remotes/origin/last-millenium-detailed -   remotes/origin/main -   remotes/origin/mid-holocene -   remotes/origin/pre-industrial -   remotes/origin/ssp585 - +More information about the available experiments and the naming scheme of the branches can also be found in the [{{ model }} configs]({{ github_configs }}) GitHub repository. + +The first step is to choose a configuration from those available.
+TODO: check correct configuration until line 106 +For example, if the required configuration is the 1° horizontal resolution with repeat-year _JRA55_ forcing (without BGC), then the branch to select is [`release-1deg_jra55_ryf`](https://github.com/ACCESS-NRI/access-om2-configs/tree/release-1deg_jra55_ryf). + +To clone this branch to a location on _Gadi_, run: + + mkdir -p ~/access-esm1.5 + cd ~/access-esm1.5 + payu clone -b expt -B release-1deg_jra55_ryf {{ github_configs }} 1deg_jra55_ryf + +In the example above the `payu clone` command clones the 1° repeat-year JRA55 configuration (` -B release-1deg_jra55_ryf`) +to a new experiment branch (`-b expt`) to a directory named `1deg_jra55_ryf`. +!!! admonition tip + Anyone using a configuration is advised to clone only a single branch (as shown in the example above) and not the entire repository. -The green-coloured branch (preceded by a star sign `*`) indicates the local branch you are currently in.
-The red-coloured branches are the available remote branches, formatted as `remotes/origin/`.
-To switch to a specific branch you can run the following command: -``` -git checkout -``` -For example, the pre-industrial configuration of {{ model }} is available in the `pre-industrial` branch. To use the pre-industrial configuration, run: -``` -git checkout pre-industrial -``` - git checkout pre-industrial - branch 'pre-industrial' set up to track 'origin/pre-industrial'. - Switched to a new branch 'pre-industrial' - git branch -   main - ٭ pre-industrial + mkdir -p ~/access-om2 + cd ~/access-om2 + payu clone -b expt -B release-1deg_jra55_ryf https://github.com/ACCESS-NRI/access-om2-configs.git 1deg_jra55_ryf + Cloned repository from https://github.com/ACCESS-NRI/access-om2-configs.git to directory: .../access-om/1deg_jra55_ryf + Created and checked out new branch: expt + laboratory path: /scratch/.../access-om2 + binary path: /scratch/.../access-om2/bin + input path: /scratch/.../access-om2/input + work path: /scratch/.../access-om2/work + archive path: /scratch/.../access-om2/archive + Updated metadata. Experiment UUID: daeee7ff-07e4-4f93-823b-cb7c6e4bdb6e + Added archive symlink to /scratch/.../access-om2/archive/1deg_jra55_ryf-expt-daeee7ff + To change directory to control directory run: + cd 1deg_jra55_ryf -## Edit {{ model }} configuration +!!! tip + _payu_ uses branches to differentiate between different experiments in the same local git repository.
+ For this reason, it is recommended to always set the cloned branch name (`expt` in the example above) to something meaningful for the planned experiment.
+ For more information refer to this [payu tutorial](https://forum.access-hive.org.au/t/access-om2-payu-tutorial/1750#select-experiment-12). -It is good practice to create a new git branch to store all your modifications for a particular run, so as not to modify the reference configuration. +---------------------------------------------------------------------------------------- -To create a new branch called _example_run_, as a copy of the `pre-industrial` branch, from within the `access-esm` directory execute: -``` -git checkout -b example_run --no-track origin/pre-industrial -``` +## Run {{ model }} configuration +TODO: modify file paths for ACCESS-ESM +If you want to modify your configuration, refer to [Edit {{ model }} configuration](#edit-{{ model.lower() }}-configuration). + +{{ model }} configurations run on [_Gadi_][gadi] through a [PBS job][PBS job] submission managed by [_payu_][payu]. + +The general layout of a _payu_-supported model run consists of two main directories: + +- The _control_ directory contains the model configuration and serves as the execution directory for running the model (in this example, the cloned directory `~/access-om2/1deg_jra55_ryf`). +- The _laboratory_ directory, where all the model components reside. For {{ model }}, it is typically `/scratch/$PROJECT/$USER/access-om2`. + +This separates the small text configuration files from the larger binary outputs and inputs. In this way, the _control_ directory can be in the `$HOME` directory (as it is the only filesystem actively backed-up on _Gadi_). The quotas for `$HOME` are low and strict, which limits what can be stored there, so it is not suitable for larger files. + +The _laboratory_ directory is a shared space for all _payu_ experiments using the same model.
+Inside the _laboratory_ directory there are two subdirectories: + +- `work` → a directory where _payu_ automatically creates a temporary subdirectory while the model is run. The temporary subdirectory gets created as part of a run and then removed after the run succeeds. +- `archive` → the directory where the output is stored following each successful run. + +Within each of the above directories _payu_ automatically creates subdirectories uniquely named according to the experiment being run.
+_Payu_ also creates symbolic links in the _control_ directory pointing to the `archive` and `work` directories. + +This design allows multiple self-resubmitting experiments that share common executables and input data to be run simultaneously. + +!!! admonition warning + The _laboratory_ directory might get deleted if not accessed for several days. To know more about how to preserve your experiment refer to [Syncing output data](#syncing-output-data). + +### Run configuration + +To run {{ model }} configuration execute the following command from within the *control* directory: + + payu run + +This will submit a single job to the queue with a run length of `restart_period`.
+For information about `restart_period`, refer to [Change run length](#change-run-length). -This command will also switch to the new `example_run` branch. - git branch -   main - ٭ pre-industrial - git checkout -b example_run --no-track origin/pre-industrial - Switched to a new branch 'example_run' - git branch - ٭ example_run -   main -   pre-industrial + cd ~/access-om2/1deg_jra55_ryf + payu run + payu: warning: Job request includes 47 unused CPUs. + payu: warning: CPU request increased from 241 to 288 + Loading input manifest: manifests/input.yaml + Loading restart manifest: manifests/restart.yaml + Loading exe manifest: manifests/exe.yaml + payu: Found modules in /opt/Modules/v4.3.0 + + qsub -q normal -P tm70 -l walltime=10800 -l ncpus=288 -l mem=1000GB -N 1deg_jra55_ryf -l wd -j n -v PAYU_PATH=/g/data/hh5/public/apps/miniconda3/envs/analysis3-23.10/bin,MODULESHOME=/opt/Modules/v4.3.0,MODULES_CMD=/opt/Modules/v4.3.0/libexec/modulecmd.tcl,MODULEPATH=/g/data/hr22/modulefiles:/g/data/hh5/public/modules:/etc/scl/modulefiles:/opt/Modules/modulefiles:/opt/Modules/v4.3.0/modulefiles:/apps/Modules/modulefiles -W umask=027 -l storage=gdata/hh5+gdata/vk83 -- /g/data/hh5/public/apps/miniconda3/envs/analysis3-23.10/bin/python3.10 /g/data/hh5/public/apps/miniconda3/envs/analysis3-23.10/bin/payu-run + + <job-ID>.gadi-pbs> +!!! tip + You can add the `-f` option to `payu run` to let the model run even if there is an existing non-empty `work` directory, created from a previous failed run or from running `payu setup`. -### Payu - -[_Payu_](https://payu.readthedocs.io/en/latest) is a workflow management tool for running numerical models in supercomputing environments.
-The general layout of a _payu_-supported model run consists of two main directories: +---------------------------------------------------------------------------------------- -- The **laboratory** directory, where all the model components reside. For {{ model }}, it is typically `/scratch/$PROJECT/$USER/access-esm`. -- The **control** directory, where the model configuration resides and from where the model is run (in this example, the cloned directory `~/access-esm`). +## Monitor {{ model }} runs -This distinction of directories separates the small-size configuration files from the larger binary outputs and inputs. In this way, the configuration files can be placed in the `$HOME` directory (as it is the only filesystem actively backed-up on _Gadi_), without overloading it with too much data. -Furthermore, this separation allows multiple self-resubmitting experiments that share common executables and input data to be run simultaneously. +The `payu run` command prints out the PBS `job-ID` (formatted as `<9-digit-number>.gadi-pbs`), as the last line to the terminal.
+To print out information on the status of a specific job, you can execute the following command: +``` +qstat +``` + + qstat <job-ID> + Job id                Name             User             Time Use S Queue + --------------------- ---------------- ---------------- -------- - ----- + <job-ID>.gadi-pbs  1deg_jra55_ryf   <$USER>            <time> R normal-exec + -To set up the _laboratory_ directory, run the following command from within the _control_ directory: +To show the status of all your submitted [PBS jobs][PBS job], you can execute the following command: ``` -payu init +qstat -u $USER ``` -This creates the _laboratory_ directory, together with relevant subdirectories, depending on the configuration. The main subdirectories of interest are: -- `work` → a temporary directory where the model is run. It gets cleaned after each run. -- `archive` → the directory where output is stored after each run. - - cd ~/access-esm/esm-pre-industrial - payu init - laboratory path: /scratch/$PROJECT/$USER/access-esm - binary path: /scratch/$PROJECT/$USER/access-esm/bin - input path: /scratch/$PROJECT/$USER/access-esm/input - work path: /scratch/$PROJECT/$USER/access-esm/work - archive path: /scratch/$PROJECT/$USER/access-esm/archive + qstat -u $USER + Job id                Name             User             Time Use S Queue + --------------------- ---------------- ---------------- -------- - ----- + <job-ID>.gadi-pbs   1deg_jra55_ryf   <$USER>            <time> R normal-exec + <job-ID>.gadi-pbs   <other-job-name> <$USER>            <time> R normal-exec + <job-ID>.gadi-pbs   <other-job-name> <$USER>            <time> R normal-exec -### Edit the _Master Configuration_ file +The default name of your job is the name of the _payu_ _control_ directory (`1deg_jra55_ryf` in the example above).
+This can be changed by altering the `jobname` in the [PBS resources section](#modify-pbs-resources) of the `config.yaml` file. -The `config.yaml` file located in the _control_ directory, is the _Master Configuration_ file.
-This file, which controls the general model configuration, contains several parts: +_S_ indicates the status of your run, where: -- **PBS resources**{: id="pbs-resources"}
- ```yaml - jobname: pre-industrial - queue: normal - walltime: 3:10:00 - ``` - These lines can be edited to change the [PBS directives](https://opus.nci.org.au/display/Help/PBS+Directives+Explained) for the [PBS job].
- For example, to run {{ model }} under the `tm70` project (ACCESS-NRI), add the following line: - ``` - project: tm70 - ``` +- _Q_ → Job waiting in the queue to start +- _R_ → Job running +- _E_ → Job ending +- _H_ → Job on hold - !!! warning - The `project` entry should always refer to a project with allocated _Service Units (SU)_, that you are a member of. If not set explicitly, {{ model }} will run using your [default project](/getting_started/first_steps#change-default-project-on-gadi) (this default project still needs to have allocated SU). For more information, check [how to join relevant NCI projects](/getting_started/first_steps#join-relevant-nci-projects). +If there are no jobs listed with your `jobname` (or if no job is listed), your run either successfully completed or was terminated due to an error.
+For more information, check [NCI documentation](https://opus.nci.org.au/display/Help/FAQ+1%3A+Why+My+Jobs+are+NOT+Running). -- **Link to the laboratory directory**
- ```yaml - # note: if laboratory is relative path, it is relative to /scratch/$PROJECT/$USER - laboratory: access-esm - ``` - These lines set the laboratory directory path, which is relative to `/scratch/$PROJECT/$USER`. Absolute paths can also be specified. +### Stop a run -- **Model**
- ```yaml - model: access - ``` - This line tells _payu_ which driver to use for the main model (`access` refers to {{ model }}). - -- **Submodels**
- - ??? code "Expand to show the full `submodels` section" - ```yaml - submodels: - - name: atmosphere - model: um - ncpus: 192 - exe: /g/data/access/payu/access-esm/bin/coe/um7.3x - input: - - /g/data/access/payu/access-esm/input/pre-industrial/atmosphere - - /g/data/access/payu/access-esm/input/pre-industrial/start_dump
- - name: ocean - model: mom - ncpus: 180 - exe: /g/data/access/payu/access-esm/bin/coe/mom5xx - input: - - /g/data/access/payu/access-esm/input/pre-industrial/ocean/common - - /g/data/access/payu/access-esm/input/pre-industrial/ocean/pre-industrial
- - name: ice - model: cice - ncpus: 12 - exe: /g/data/access/payu/access-esm/bin/coe/cicexx - input: - - /g/data/access/payu/access-esm/input/pre-industrial/ice
- - name: coupler - model: oasis - ncpus: 0 - input: - - /g/data/access/payu/access-esm/input/pre-industrial/coupler - ``` - - {{ model }} is a coupled model deploying multiple submodels (i.e. [model components](/models/configurations/access-esm/#model-components)). - This section specifies the submodels and configuration options required to execute the model correctly.
- Each submodel contains additional configuration options that are read in when the submodel is running. These options are specified in the subfolder of the _control_ directory, whose name matches the submodel's `name` (e.g., configuration options for the `atmosphere` submodel are in the `~/access-esm/esm-pre-industrial/atmosphere` directory). +If you want to manually terminate a run, you can do so by executing: +``` +qdel +``` +which kills the specified job without waiting for it to complete. -- **Collate**
- ```yaml - collate: - exe: /g/data/access/payu/access-esm/bin/mppnccombine - restart: true - mem: 4GB - ``` - The `collate` process combines a number of smaller files, which contain different parts of the model grid, into target output files. Restart files are typically tiled in the same way and will also be combined together if the `restart` option is set to `true`. +!!! tip + If you started an {{ model }} run using the `-n` option (e.g., to [run the model for more than 5 years](#run-configuration-for-more-than-5-years)), but subsequently decide not to keep running after the current process completes, you can create a file called `stop_run` in the _control_ directory.
+ This will prevent _payu_ from submitting another job. -- **Restart**
- ```yaml - restart: /g/data/access/payu/access-esm/restart/pre-industrial - ``` - This is the location of the files used for a warm restart. - -- **Start date and run length**{: id="runtime"}
- ```yaml - calendar: - start: - year: 101 - month: 1 - days: 1 - runtime: - years: 1 - months: 0 - days: 0 - ``` - This section specifies the start date and run length. - - !!! warning - The _run length_ (controlled by [`runtime`](#runtime)) can be different from the _total experiment length_. Also, while `runtime` can be reduced, it should not be increased to more than 1 year to avoid errors. For more information about the difference between _run length_ and _total experiment length_, or how to run the model for more than 1 year, refer to the section [Run configuration for multiple years](#run-configuration-for-multiple-years). +### Error and output log files -- **Number of runs per PBS submission**{: id="runspersub"}
- ```yaml - runspersub: 1 - ``` - {{ model }} configurations are often run in multiple steps (or cycles), with _payu_ running a maximum of `runspersub` runs for every [PBS job] submission. - - !!! warning - If you increase `runspersub`, you may need to increase the `walltime` in the [PBS resources](#pbs-resources). +#### PBS output files +When the model completes a run, PBS writes the standard output and error streams to two files inside the _control_ directory: `.o` and `.e`, respectively. -To find out more about other configuration settings for the `config.yaml` file, check out [how to configure your experiment with _payu_](https://payu.readthedocs.io/en/latest/config.html). +These files usually contain logs about _payu_ tasks, and give an overview of the resources used by the job.
+To move these files to the `archive` directory, use the following commmand: +``` +payu sweep +``` -### Edit a single {{ model }} component configuration +#### Model log files -Each of [{{ model }} components](/models/configurations/access-esm/#model-components) contains additional configuration options that are read in when the model component is running.
-These options are typically useful to modify the physics used in the model, the input data, or the model variables saved in the output files. +While the model is running, _payu_ saves the model standard output and error streams in the `access-om2.out` and `access-om2.err` files inside the _control_ directory, respectively.
+You can examine the contents of these files to check on the status of a run as it progresses (or after a failed run has completed). -These configuration options are specified in files located inside a subfolder of the _control_ directory, named according to the submodel's `name` specified in the `config.yaml` `submodels` section (e.g., configuration options for the `atmosphere` submodel are in the `~/access-esm/esm-pre-industrial/atmosphere` directory).
-To modify these options please refer to the User Guide of the respective model component. +!!! warning + At the end of a successful run these log files are archived to the `archive` directory and will no longer be found in the _control_ directory. If they remain in the _control_ directory after the PBS job for a run has completed it means the run has failed. -## Run {{ model }} configuration +### Model Live Diagnostics -After editing the configuration, you are ready to run {{ model }}.
-{{ model }} suites run on [_Gadi_](https://opus.nci.org.au/display/Help/0.+Welcome+to+Gadi#id-0.WelcometoGadi-Overview) through a [PBS job] submission managed by _payu_. +ACCESS-NRI developed the [Model Live Diagnostics](/model_evaluation/model_diagnostics) framework to check, monitor, visualise, and evaluate model behaviour and progress of ACCESS models currently running on _Gadi_.
+For a complete documentation on how to use this framework, check the [Model Diagnostics documentation](https://med-live-diagnostics.readthedocs.io/en/latest/index.html). -### Payu setup (optional) +### Trouble-shooting -As a first step, from within the _control_ directory, it is good practice to run: -``` -payu setup -``` +If _payu_ doesn't run correctly for some reason, a good first step is to run the following command from within the _control_ directory: + + payu setup + +This command will: + + - create the _laboratory_ and `work` directories based on the experiment configuration + - generate manifests + - report useful information to the user, such as the location of the _laboratory_ where the `work` and `archive` directories are located -This will prepare the model run, based on the experiment configuration. payu setup - laboratory path: /scratch/$PROJECT/$USER/access-esm - binary path: /scratch/$PROJECT/$USER/access-esm/bin - input path: /scratch/$PROJECT/$USER/access-esm/input - work path: /scratch/$PROJECT/$USER/access-esm/work - archive path: /scratch/$PROJECT/$USER/access-esm/archive + laboratory path: /scratch/$PROJECT/$USER/access-om2 + binary path: /scratch/$PROJECT/$USER/access-om2/bin + input path: /scratch/$PROJECT/$USER/access-om2/input + work path: /scratch/$PROJECT/$USER/access-om2/work + archive path: /scratch/$PROJECT/$USER/access-om2/archive Loading input manifest: manifests/input.yaml Loading restart manifest: manifests/restart.yaml Loading exe manifest: manifests/exe.yaml Setting up atmosphere Setting up ocean Setting up ice - Setting up coupler + Setting up access-om2 Checking exe and input manifests Updating full hashes for 3 files in manifests/exe.yaml Creating restart manifest - Updating full hashes for 30 files in manifests/restart.yaml Writing manifests/restart.yaml Writing manifests/exe.yaml -!!! tip - This step can be skipped as it is also included in the run command. However, running it explicitly helps to check for errors and make sure executable and restart directories are accessible. +This can help to isolate issues such as permissions problems accessing files and directories, missing files or malformed/incorrect paths. -### Run configuration +---------------------------------------------------------------------------------------- -To run {{ model }} configuration for one _run length_ (controlled by [`runtime`](#runtime) in the `config.yaml` file), from within the _control_ directory execute: +## {{ model }} outputs -``` -payu run -f -``` +At the end of a successful model run, output files, restart files and log files are moved from the `work` directory to the `archive` directory.
+Symbolic links to these directories are also provided in the _control_ directory for convenience. -This will submit a single job to the queue with a _total experiment length_ of `runtime`. If there is no previous run, it will start from the [`start`](#runtime) date indicated in the `config.yaml` file. Otherwise, it will perform a warm restart from a previously saved restart file.
+If a model run is unsuccessful, the `work` directory is left untouched to facilitate the identification of the cause of the model failure. -!!! tip - The `-f` option ensures that _payu_ will run even if there is an existing non-empty `work` directory created from a previous failed run. +Outputs and restarts are stored in subfolders within the `archive` directory, subdivided for each run of the model.
+Output and restart folders are called `outputXXX` and `restartXXX`, respectively, where _XXX_ is the run number starting from `000`. + +Model components are separated into subdirectories within the output and restart directories. - payu run -f - Loading input manifest: manifests/input.yaml - Loading restart manifest: manifests/restart.yaml - Loading exe manifest: manifests/exe.yaml - payu: Found modules in /opt/Modules/v4.3.0 - qsub -q normal -P <project> -l walltime=11400 -l ncpus=384 -l mem=1536GB -N pre-industrial -l wd -j n -v PAYU_PATH=/g/data/hh5/public/apps/miniconda3/envs/analysis3-23.01/bin,MODULESHOME=/opt/Modules/v4.3.0,MODULES_CMD=/opt/Modules/v4.3.0/libexec/modulecmd.tcl,MODULEPATH=/g/data3/hh5/public/modules:/etc/scl/modulefiles:/opt/Modules/modulefiles:/opt/Modules/v4.3.0/modulefiles:/apps/Modules/modulefiles -W umask=027 -l storage=gdata/access+gdata/hh5 -- /g/data/hh5/public/apps/miniconda3/envs/analysis3-23.01/bin/python3.9 /g/data/hh5/public/apps/miniconda3/envs/analysis3-23.01/bin/payu-run - <job-ID>.gadi-pbs + cd ~/access-om2/1deg_jra55_ryf + ls + output000 pbs_logs restart000 -### Run configuration for multiple years +---------------------------------------------------------------------------------------- + +## Edit {{ model }} configuration + +This section describes how to modify {{ model }} configuration.
+The modifications discussed in this section can change the way {{ model }} is run by _payu_, or how its specific [model components] are configured and coupled together. + +The `config.yaml` file located in the _control_ directory is the _Master Configuration_ file, which controls the general model configuration. It contains several parts, some of which it is more likely will need modification, and others which are rarely changed without having a deep understanding of how the model is configured. + +To find out more about configuration settings for the `config.yaml` file, refer to [how to configure your experiment with payu](https://payu.readthedocs.io/en/latest/config.html). + +### Change run length + +One of the most common changes is to adjust the duration of the model run.
+For example, when debugging changes to a model, it is common to reduce the run length to minimise resource consumption and return faster feedback on changes. + +The run length is controlled by the `restart_period` field in the `&date_manager_nml` section of the `~/access-om2/1deg_jra55_ryf/accessom2.nml` file: + + &date_manager_nml + forcing_start_date = '1958-01-01T00:00:00' + forcing_end_date = '2019-01-01T00:00:00'
+ ! Runtime for a single segment/job/submit, format is years, months, seconds, + ! two of which must be zero. + restart_period = 5, 0, 0 + &end -If you want to run {{ model }} configuration for multiple _run lengths_ (controlled by [`runtime`](#runtime) in the `config.yaml` file), use the option `-n`: +The format is `restart_period = , , `. +For example, to make the model run for 1 year, 4 months and 10 days, change `restart_period` to: + + restart_period = 1, 4, 10 + +!!! warning + While `restart_period` can be reduced, it should not be increased to more than 5 years to avoid errors. +

+ It is also important to differentiate between _run length_ and _total experiment length_.
+ For more information about their difference, or how to run the model for more than 5 years, refer to [Run configuration for more than 5 years](#run-configuration-for-more-than-5-years). + +#### Run configuration for more than 5 years + +As mentioned in the [Change run length](#change-run-length) section, you cannot specify more than 5 years as `restart_period`.
+If you want to run a configuration for more than 5 years, you need to use the `-n` option: + + payu run -n + +This will run {{ model }} `number-of-runs` consecutive times, each with a *run length* equal to `restart_period`.
+This way, the *total experiment length* will be `restart_period * number-of-runs`. + +For example, to run a configuration for a total of 50 years with a `restart_period` of 5 years, the `number-of-runs` should be set to `10`: + + payu run -n 10 + +### Modify PBS resources + +If the model has been altered and needs more time to complete, more memory, or needs to be submitted under a different NCI project, you will need to modify the following section in the `config.yaml`: + +```yaml +# If submitting to a different project to your default, uncomment line below +# and replace PROJECT_CODE with appropriate code. This may require setting shortpath +# project: PROJECT_CODE + +# Force payu to always find, and save, files in this scratch project directory +# shortpath: /scratch/PROJECT_CODE + +queue: normal +walltime: 3:00:00 +jobname: 1deg_jra55_ryf +mem: 1000GB ``` -payu run -f -n + +These lines can be edited to change the [PBS directives](https://opus.nci.org.au/display/Help/PBS+Directives+Explained) for the [PBS job][PBS job]. + +For example, to run {{ model }} under the `ol01` project (COSIMA Working Group), uncomment the line beginning with `# project` by deleting the `#` symbol and replace `PROJECT_CODE` wih `ol01`: + +```yaml +project: ol01 ``` -This will run the configuration `number-of-runs` times with a _total experiment length_ of `runtime * number-of-runs`. The number of consecutive [PBS jobs][PBS job] submitted to the queue depends on the [`runspersub`](#runspersub) value specified in the config.yaml file. +!!! warning + If projects other than `ol01` are used to run {{ model }} configuration, then the `shortpath` field also needs to be uncommented and the path to the desired `/scratch/PROJECT_CODE` added.
+ Doing this will make sure the same `/scratch` location is used for the _laboratory_, regardless of which project is used to run the experiment. +

+ To run {{ model }}, you need to be a member of a project with allocated _Service Units (SU)_. For more information, check [how to join relevant NCI projects](/getting_started/first_steps#join-relevant-nci-projects). + +### Syncing output data + +The _laboratory_ directory is typically under the `/scratch` storage on _Gadi_, where [files are regularly deleted once they have been unaccessed for a period of time](https://opus.nci.org.au/pages/viewpage.action?pageId=156434436). For this reason climate model outputs need to be moved to a location with longer term storage.
+On _Gadi_, this is typically under a project code on `/g/data`. + +_Payu_ has built-in support to sync outputs, restarts and a copy of the _control_ directory git history to another location.
+This feature is controlled by the following section in the `config.yaml` file: +```yaml +# Sync options for automatically copying data from ephemeral scratch space to +# longer term storage +sync: + enable: False # set path below and change to true + path: none # Set to location on /g/data or a remote server and path (rsync syntax) + exclude: + - '*.nc.*' + - 'iceh.????-??-??.nc' +``` +To enable output syncing, change `enable` to `True`, and set `path` to a location on `/g/data`. -### Understand `runtime`, `runspersub`, and `-n` parameters +### Saving model restarts -With the correct use of [`runtime`](#runtime), [`runspersub`](#runspersub) and `-n` parameters, you can have full control of your run.
+{{ model }} outputs restart files after every run to allow for subsequent runs to start from a previously saved model state.
+Restart files can occupy a significant amount of disk space, and keeping a lot of them is often not necessary. -- `runtime` defines the _run length_. -- `runspersub` defines the maximum number of runs for every [PBS job] submission. -- `-n` sets the number of runs to be performed. +The `restart_freq` field in the `config.yaml` file specifies a strategy for retaining restart files.
+This can either be a number (in which case every _nth_ restart file is retained), or one of the following pandas-style datetime frequencies: -Now some practical examples: +- `YS` → start of the year +- `MS` → start of the month +- `D` → day +- `H` → hour +- `T` → minute +- `S` → second -- **Run 20 years of simulation with resubmission every 5 years**
- To have a _total experiment length_ of 20 years with a 5-year resubmission cycle, leave [`runtime`](#runtime) as the default value of `1 year` and set [`runspersub`](#runspersub) to `5`. Then, run the configuration with `-n` set to `20`: - ``` - payu run-f -n 20 - ``` - This will submit subsequent jobs for the following years: 1 to 5, 6 to 10, 11 to 15, and 16 to 20, which is a total of 4 PBS jobs. +For example, to preserve the ability to restart {{ model }} every 50 model-years, set: +```yaml +restart_freq: '50YS' +``` + +The most recent sequential restarts are retained, and only deleted after a permanently archived restart file has been produced. + +For more information, check [_payu_ Configuration Settings documentation](https://payu.readthedocs.io/en/latest/config.html#model). + +### Other configuration options + +!!! warning + The following sections in the `config.yaml` file control configuration options that are rarely modified, and often require a deeper understanding of how {{ model }} is structured to be safely changed. + +#### Model configuration + +This section tells _payu_ which driver to use for the main model configuration (`access-om2`) and the location of all inputs that are common to all its [model components]. + +```yaml +name: common +model: access-om2 +input: /g/data/ik11/inputs/access-om2/input_20201102/common_1deg_jra55 +``` +The `name` field is not actually used for the configuration run, so it can be safely ignored. + +#### Submodels + +{{ model }} is a coupled model deploying multiple submodels (i.e. [model components]). + +This section specifies the submodels and configuration options required to execute {{ model }} correctly. + +Each submodel contains additional configuration options that are read in when the submodel is running. These options are specified in the subfolder of the _control_ directory whose name matches the submodel's `name` (e.g., configuration options for the `ocean` submodel are in the `~/access-om2/1deg_jra55_ryf/ocean` directory). + +??? code "Expand to show the full `submodels` section" -- **Run 7 years of simulation with resubmission every 3 years**
- To have a _total experiment length_ of 7 years with a 3-year resubmission cycle, leave [`runtime`](#runtime) as the default value of `1 year` and set [`runspersub`](#runspersub) to `3`. Then, run the configuration with `-n` set to `7`: - ``` - payu run -f -n 7 - ``` - This will submit subsequent jobs for the following years: 1 to 3, 4 to 6, and 7, which is a total of 3 PBS jobs. - -- **Run 3 months and 10 days of simulation in a single submission**
- To have a _total experiment length_ of 3 months and 10 days in a single submission, set the [`runtime`](#runtime) as follows: - ```yaml - years: 0 - months: 3 - days: 10 - ``` - Set [`runspersub`](#runspersub) to `1` (or any value > 1) and run the configuration without `-n` option (or with `-n` set to `1`): - ``` - payu run -f - ``` - -- **Run 1 year and 4 months of simulation with resubmission every 4 months**
- To have a _total experiment length_ of 1 year and 4 months (16 months), you need to split it into multiple runs. For example, 4 runs of 4 months each. In this case, set the [`runtime`](#runtime) as follows: ```yaml - years: 0 - months: 4 - days: 0 - ``` - Since the _run length_ is set to 4 months, set [`runspersub`](#runspersub) to `1` to resubmit your jobs every 4 months (i.e. every run). Then, run the configuration with `-n` set to `4`: - ``` - payu run -f -n 4 + submodels: + - name: atmosphere + model: yatm + exe: /g/data/vk83/apps/spack/0.20/release/linux-rocky8-x86_64/intel-19.0.5.281/libaccessom2-git.2023.10.26=2023.10.26-ieiy3e7hidn4dzaqly3ly2yu45mecgq4/bin/yatm.exe + input: + - /g/data/vk83/experiments/inputs/access-om2/remapping_weights/JRA55/global.1deg/2020.05.30/rmp_jrar_to_cict_CONSERV.nc + - /g/data/vk83/experiments/inputs/JRA-55/RYF/v1-4/data + ncpus: 1 + + - name: ocean + model: mom + exe: /g/data/vk83/apps/spack/0.20/release/linux-rocky8-x86_64/intel-19.0.5.281/mom5-git.2023.11.09=2023.11.09-ewcdbrfukblyjxpkhd3mfkj4yxfolal4/bin/fms_ACCESS-OM.x + input: + - /g/data/vk83/experiments/inputs/access-om2/ocean/grids/mosaic/global.1deg/2020.05.30/grid_spec.nc + - /g/data/vk83/experiments/inputs/access-om2/ocean/grids/mosaic/global.1deg/2020.05.30/ocean_hgrid.nc + - /g/data/vk83/experiments/inputs/access-om2/ocean/grids/mosaic/global.1deg/2020.05.30/ocean_mosaic.nc + - /g/data/vk83/experiments/inputs/access-om2/ocean/grids/bathymetry/global.1deg/2020.10.22/topog.nc + - /g/data/vk83/experiments/inputs/access-om2/ocean/grids/bathymetry/global.1deg/2020.10.22/ocean_mask.nc + - /g/data/vk83/experiments/inputs/access-om2/ocean/grids/vertical/global.1deg/2020.10.22/ocean_vgrid.nc + - /g/data/vk83/experiments/inputs/access-om2/ocean/processor_masks/global.1deg/216.16x15/2020.05.30/ocean_mask_table + - /g/data/vk83/experiments/inputs/access-om2/ocean/chlorophyll/global.1deg/2020.05.30/chl.nc + - /g/data/vk83/experiments/inputs/access-om2/ocean/initial_conditions/global.1deg/2020.10.22/ocean_temp_salt.res.nc + - /g/data/vk83/experiments/inputs/access-om2/ocean/tides/global.1deg/2020.05.30/tideamp.nc + - /g/data/vk83/experiments/inputs/access-om2/ocean/tides/global.1deg/2020.05.30/roughness_amp.nc + - /g/data/vk83/experiments/inputs/access-om2/ocean/tides/global.1deg/2020.05.30/roughness_cdbot.nc + - /g/data/vk83/experiments/inputs/access-om2/ocean/surface_salt_restoring/global.1deg/2020.05.30/salt_sfc_restore.nc + ncpus: 216 + + - name: ice + model: cice5 + exe: /g/data/vk83/apps/spack/0.20/release/linux-rocky8-x86_64/intel-19.0.5.281/cice5-git.2023.10.19=2023.10.19-rh3xfkrgajya3ghtliacuhlx3pgvrzqs/bin/cice_auscom_360x300_24x1_24p.exe + input: + - /g/data/vk83/experiments/inputs/access-om2/ice/grids/global.1deg/2020.05.30/grid.nc + - /g/data/vk83/experiments/inputs/access-om2/ice/grids/global.1deg/2020.10.22/kmt.nc + - /g/data/vk83/experiments/inputs/access-om2/ice/initial_conditions/global.1deg/2020.05.30/i2o.nc + - /g/data/vk83/experiments/inputs/access-om2/ice/initial_conditions/global.1deg/2020.05.30/o2i.nc + - /g/data/vk83/experiments/inputs/access-om2/ice/initial_conditions/global.1deg/2020.05.30/u_star.nc + - /g/data/vk83/experiments/inputs/access-om2/ice/initial_conditions/global.1deg/2020.05.30/monthly_sstsss.nc + ncpus: 24 ``` -## Monitor {{ model }} runs +#### Collate -You can execute the following command to show the status of all your submitted [PBS jobs][PBS job]: +Rather than outputting a single diagnostic file over the whole model horizontal grid, [MOM](/models/model_components/ocean/#modular-ocean-model-mom) typically generates diagnostic outputs as tiles, each of which spans over a portion of the model horizontal grid. +The `collate` section in the `yaml.conf` file controls the process that combines these smaller files into a single output file. +```yaml +collate: + restart: true + walltime: 1:00:00 + mem: 30GB + ncpus: 4 + queue: normal + exe: /g/data/ik11/inputs/access-om2/bin/mppnccombine ``` -qstat -u $USER -``` - - qstat -u $USER - Job id                Name             User             Time Use S Queue - --------------------- ---------------- ---------------- -------- - ----- - <job-ID>.gadi-pbs     pre-industrial   <$USER>            <time> R normal-exec - <job-ID-2>.gadi-pbs   <other-job-name> <$USER>            <time> R normal-exec - <job-ID-3>.gadi-pbs   <other-job-name> <$USER>            <time> R normal-exec - +Restart files are typically tiled in the same way and will also be combined together if the `restart` field is set to `true`. -If you changed the `jobname` in the [PBS resources](#pbs-resources) of the _Master Configuration_ file, that will appear as your job's _Name_ instead of `pre-industrial`.
+#### Runlog -_S_ indicates the status of your run, where: - -- _Q_ → Job waiting in the queue to start -- _R_ → Job running -- _E_ → Job ending -- _H_ → Job on hold +```yaml +runlog: true +``` +When running a new configuration, _payu_ automatically commits changes with `git` if `runlog` is set to `true`. -If there are no jobs listed with your `jobname` (or if no job is listed), your run either successfully completed or was terminated due to an error.
-For more information, check [NCI documentation](https://opus.nci.org.au/display/Help/FAQ+1%3A+Why+My+Jobs+are+NOT+Running). +!!! warning + This should not be changed as it is an essential part of the provenance of an experiment.
+ _payu_ updates the manifest files for every run, and relies on `runlog` to save this information in the `git` history, so there is a record of all inputs, restarts, and executables used in an experiment. -### Stop a run +#### Platform -If you want to manually terminate a run, you can do so by executing: +```yaml +platform: + nodesize: 48 ``` -qdel +Set platform-specific default parameters.
+In the example above, the default number of cpus per node is set to 48. +!!! warning + This might need changing if the configuration is run on hardware with different node structure. + +#### Userscripts + +```yaml +userscripts: + error: tools/resub.sh + run: rm -f resubmit.count + sync: /g/data/vk83/apps/om2-scripts/concatenate_ice/concat_ice_daily.sh ``` -### Error and output log files +A dictionary to run scripts or subcommands at various stages of a _payu_ submission. -While the model is running, _payu_ saves the standard output and standard error in the respective `access.out` and `access.err` files in the _control_ directory. You can examine the contents of these files to check on the status of a run as it progresses.
-When the model completes its run, or if it crashes, the output and error log files are by default renamed as `jobname.o` and `jobname.e`, respectively. +- `error` gets called if the model does not run correctly and returns an error code. +- `run` gets called after the model execution, but prior to model output archive. +- `sync` gets called at the start of the sync pbs job. -### Model Live Diagnostics +For more information about specific `userscripts` fields, check the relevant section of [_payu_ Configuration Settings documentation](https://payu.readthedocs.io/en/latest/config.html#postprocessing). -ACCESS-NRI developed the [Model Live Diagnostics](/model_evaluation/model_diagnostics) framework to check, monitor, visualise, and evaluate model behaviour and progress of ACCESS models currently running on _Gadi_.
-For a complete documentation on how to use this framework, check the [Model Diagnostics documentation](https://med-live-diagnostics.readthedocs.io/en/latest/index.html). +#### Miscellaneous -## {{ model }} outputs +The following configuration settings should never require changing: -At the end of the model run, output files (and restart files) are moved from the `work` directory into the `archive` directory under `/scratch/$PROJECT/$USER/access-esm/archive/access-esm`, where they are further subdivided for each run. They are also symlinked in the _control_ directory to `~/access-esm/archive`.
-The naming format for a typical output folder is `outputXXX` and for a restart folder `restartXXX`, where _XXX_ is the run number starting from `000`. +```yaml +stacksize: unlimited +mpirun: --mca io ompio --mca io_ompio_num_aggregators 1 +qsub_flags: -W umask=027 +env: + UCX_LOG_LEVEL: 'error' +``` -!!! tip - A run with a different {{ model }} configuration (different git branch) counts as a new run.
- Thus, if output folders already exist, the internal number of the new output folder will be set to the first available _XXX_ number. +### Edit a single {{ model }} component configuration -Outputs and restarts are separated in the respective folders for each model component.
-For the atmospheric output data, the files are usually [UM fieldsfile](https://code.metoffice.gov.uk/doc/um/latest/papers/umdp_F03.pdf), formatted as `a.p`. - - cd /scratch/$PROJECT/$USER/access-esm/archive/esm-pre-industrial - ls - output000 pbs_logs restart000 - ls output000/atmosphere - aiihca.daa1210 aiihca.daa1810 aiihca.paa1apr aiihca.paa1jun aiihca.pea1apr aiihca.pea1jun aiihca.pga1apr aiihca.pga1jun atm.fort6.pe0 exstat ihist prefix.CNTLGEN UAFLDS_A aiihca.daa1310 aiihca.daa1910 aiihca.paa1aug aiihca.paa1mar aiihca.pea1aug aiihca.pea1mar aiihca.pga1aug aiihca.pga1mar cable.nml fort.57 INITHIS prefix.PRESM_A um_env.py aiihca.daa1410 aiihca.daa1a10 aiihca.paa1dec aiihca.paa1may aiihca.pea1dec aiihca.pea1may aiihca.pga1dec aiihca.pga1may CNTLALL ftxx input_atm.nml SIZES xhist aiihca.daa1510 aiihca.daa1b10 aiihca.paa1feb aiihca.paa1nov aiihca.pea1feb aiihca.pea1nov aiihca.pga1feb aiihca.pga1nov CONTCNTL ftxx.new namelists STASHC aiihca.daa1610 aiihca.daa1c10 aiihca.paa1jan aiihca.paa1oct aiihca.pea1jan aiihca.pea1oct aiihca.pga1jan aiihca.pga1oct debug.root.01 ftxx.vars nout.000000 thist aiihca.daa1710 aiihca.daa2110 aiihca.paa1jul aiihca.paa1sep aiihca.pea1jul aiihca.pea1sep aiihca.pga1jul aiihca.pga1sep errflag hnlist prefix.CNTLATM UAFILES_A - +Each of [{{ model }} components][model components] contains additional configuration options that are read in when the model component is running.
+These options are typically useful to modify the physics used in the model, the input data, or the model variables saved in the output files. + +These configuration options are specified in files located inside a subfolder of the _control_ directory, named according to the submodel's `name` specified in the `config.yaml` `submodels` section (e.g., configuration options for the _ocean_ component are in the `~/access-om2/1deg_jra55_ryf/ocean` directory).
+To modify these options please refer to the User Guide of the respective model component. + +## Get Help + +If you have questions or need help regarding {{ model }}, consider creating a topic in the [COSIMA category of the ACCESS Hive Forum](https://forum.access-hive.org.au/c/cosima/29).
+For assistance on how to request help from ACCESS-NRI, follow the [guidelines on how to get help](/about/user_support/#still-need-help). +- [https://github.com/access-nri/access-esm1.5](https://github.com/access-nri/access-esm1.5) +- [https://opus.nci.org.au/](https://opus.nci.org.au/) - [https://github.com/coecms/esm-pre-industrial](https://github.com/coecms/esm-pre-industrial) - [https://payu.readthedocs.io/en/latest/usage.html](https://payu.readthedocs.io/en/latest/usage.html) diff --git a/docs/models/run-a-model/run-access-esm__old.md b/docs/models/run-a-model/run-access-esm__old.md new file mode 100644 index 000000000..6b4c74e89 --- /dev/null +++ b/docs/models/run-a-model/run-access-esm__old.md @@ -0,0 +1,470 @@ +{% set model = "ACCESS-ESM" %} +[PBS job]: https://opus.nci.org.au/display/Help/4.+PBS+Jobs + +!!! bug + There is a known issue with the {{ model }} configurations currently set up in the GitHub repository below. + {{ model }} configurations are currently being updated and an ACCESS-NRI release will come + in the following weeks. + If you encounter problems with the current guide in this page, consider asking for help on the [ACCESS-Hive Forum](https://forum.access-hive.org.au). + +# Run {{ model }} + +## Prerequisites + +### General prerequisites + +Before running {{ model }}, you need to fulfil general prerequisites outlined in the [First Steps](/getting_started/first_steps) section. + +If you are unsure whether {{ model }} is the right choice for your experiment, take a look at the overview of [ACCESS Models](/models). + +### Model-specific prerequisites + +{{ model }} is installed on NCI's supercomputer _Gadi_ and uses [_payu_](#payu), a tool for running and managing model experiments. Following these prerequisites ensures you have access to this infrastructure. + +- **Join the _access_ and _hh5_ projects at NCI**
+ To join these projects, request membership on the respective [access](https://my.nci.org.au/mancini/project/access/join) and [hh5](https://my.nci.org.au/mancini/project/hh5/join) NCI project pages.
+ For more information on joining specific NCI projects, refer to [How to connect to a project](https://opus.nci.org.au/display/Help/How+to+connect+to+a+project). +- **Payu**
+ _Payu_ on _Gadi_ is available through the `conda/analysis3` environment in the _hh5_ project.
+ After obtaining _hh5_ project membership, load the `conda/analysis3` environment to automatically retrieve _payu_ as follows: + ``` + module use /g/data/hh5/public/modules + module load conda/analysis3 + ``` + To check that _payu_ is available, run: + ``` + payu --version + ``` + + payu --version + 1.0.19 + + +## Get {{ model }} configuration + +{{ model }} configurations are available in the [coecms GitHub](https://github.com/coecms/access-esm) repository.
+To get it on _Gadi_, clone the {{ model }} GitHub repository by running: +``` +git clone https://github.com/coecms/access-esm.git +``` +This will create the `access-esm` folder. + + git clone https://github.com/coecms/access-esm.git + Cloning into 'access-esm'... + remote: Enumerating objects: 1625, done. + remote: Counting objects: 100% (1625/1625), done. + remote: Compressing objects: 100% (575/575), done. + remote: Total 1625 (delta 1042), reused 1621 (delta 1040), pack-reused 0 + Receiving objects: 100% (1625/1625), 2.79 MiB | 11.24 MiB/s, done. + Resolving deltas: 100% (1042/1042), done. + ls ~/access-esm + atmosphere config.yaml coupler ice manifests ocean README.md + + +!!! warning + Some modules may interfere with `git` commands (e.g., matlab/R2018a). If you have trouble cloning the repository, run the following command before trying again: + ``` + module purge + ``` + After this step, don't forget to reload the `conda/analysis3` module to retrieve _payu_, as specified in the [Model-specific prerequisites](#model-specific-prerequisites) section. + +Each {{ model }} configuration is stored as a specially-named branch in the {{ model }} GitHub repository. +To check all the available branches on the repo, run the following command inside the newly-created `access-esm` folder: +``` +git branch -a +``` + + cd ~/access-esm + git branch -a + ٭ main +   remotes/origin/82ka +   remotes/origin/HEAD → origin/main +   remotes/origin/ccarouge-patch-1 +   remotes/origin/historical +   remotes/origin/last-interglacial +   remotes/origin/last-millenium +   remotes/origin/last-millenium-detailed +   remotes/origin/main +   remotes/origin/mid-holocene +   remotes/origin/pre-industrial +   remotes/origin/ssp585 + + +The green-coloured branch (preceded by a star sign `*`) indicates the local branch you are currently in.
+The red-coloured branches are the available remote branches, formatted as `remotes/origin/`.
+To switch to a specific branch you can run the following command: +``` +git checkout +``` +For example, the pre-industrial configuration of {{ model }} is available in the `pre-industrial` branch. To use the pre-industrial configuration, run: +``` +git checkout pre-industrial +``` + + git checkout pre-industrial + branch 'pre-industrial' set up to track 'origin/pre-industrial'. + Switched to a new branch 'pre-industrial' + git branch +   main + ٭ pre-industrial + + +## Edit {{ model }} configuration + +It is good practice to create a new git branch to store all your modifications for a particular run, so as not to modify the reference configuration. + +To create a new branch called _example_run_, as a copy of the `pre-industrial` branch, from within the `access-esm` directory execute: +``` +git checkout -b example_run --no-track origin/pre-industrial +``` + +This command will also switch to the new `example_run` branch. + + git branch +   main + ٭ pre-industrial + git checkout -b example_run --no-track origin/pre-industrial + Switched to a new branch 'example_run' + git branch + ٭ example_run +   main +   pre-industrial + + +### Payu + +[_Payu_](https://payu.readthedocs.io/en/latest) is a workflow management tool for running numerical models in supercomputing environments.
+The general layout of a _payu_-supported model run consists of two main directories: + +- The **laboratory** directory, where all the model components reside. For {{ model }}, it is typically `/scratch/$PROJECT/$USER/access-esm`. +- The **control** directory, where the model configuration resides and from where the model is run (in this example, the cloned directory `~/access-esm`). + +This distinction of directories separates the small-size configuration files from the larger binary outputs and inputs. In this way, the configuration files can be placed in the `$HOME` directory (as it is the only filesystem actively backed-up on _Gadi_), without overloading it with too much data. +Furthermore, this separation allows multiple self-resubmitting experiments that share common executables and input data to be run simultaneously. + +To set up the _laboratory_ directory, run the following command from within the _control_ directory: +``` +payu init +``` +This creates the _laboratory_ directory, together with relevant subdirectories, depending on the configuration. The main subdirectories of interest are: + +- `work` → a temporary directory where the model is run. It gets cleaned after each run. +- `archive` → the directory where output is stored after each run. + + + cd ~/access-esm/esm-pre-industrial + payu init + laboratory path: /scratch/$PROJECT/$USER/access-esm + binary path: /scratch/$PROJECT/$USER/access-esm/bin + input path: /scratch/$PROJECT/$USER/access-esm/input + work path: /scratch/$PROJECT/$USER/access-esm/work + archive path: /scratch/$PROJECT/$USER/access-esm/archive + + +### Edit the _Master Configuration_ file + +The `config.yaml` file located in the _control_ directory, is the _Master Configuration_ file.
+This file, which controls the general model configuration, contains several parts: + +- **PBS resources**{: id="pbs-resources"}
+ ```yaml + jobname: pre-industrial + queue: normal + walltime: 3:10:00 + ``` + These lines can be edited to change the [PBS directives](https://opus.nci.org.au/display/Help/PBS+Directives+Explained) for the [PBS job].
+ For example, to run {{ model }} under the `tm70` project (ACCESS-NRI), add the following line: + ``` + project: tm70 + ``` + + !!! warning + The `project` entry should always refer to a project with allocated _Service Units (SU)_, that you are a member of. If not set explicitly, {{ model }} will run using your [default project](/getting_started/first_steps#change-default-project-on-gadi) (this default project still needs to have allocated SU). For more information, check [how to join relevant NCI projects](/getting_started/first_steps#join-relevant-nci-projects). + +- **Link to the laboratory directory**
+ ```yaml + # note: if laboratory is relative path, it is relative to /scratch/$PROJECT/$USER + laboratory: access-esm + ``` + These lines set the laboratory directory path, which is relative to `/scratch/$PROJECT/$USER`. Absolute paths can also be specified. + +- **Model**
+ ```yaml + model: access + ``` + This line tells _payu_ which driver to use for the main model (`access` refers to {{ model }}). + +- **Submodels**
+ + ??? code "Expand to show the full `submodels` section" + ```yaml + submodels: + - name: atmosphere + model: um + ncpus: 192 + exe: /g/data/access/payu/access-esm/bin/coe/um7.3x + input: + - /g/data/access/payu/access-esm/input/pre-industrial/atmosphere + - /g/data/access/payu/access-esm/input/pre-industrial/start_dump
+ - name: ocean + model: mom + ncpus: 180 + exe: /g/data/access/payu/access-esm/bin/coe/mom5xx + input: + - /g/data/access/payu/access-esm/input/pre-industrial/ocean/common + - /g/data/access/payu/access-esm/input/pre-industrial/ocean/pre-industrial
+ - name: ice + model: cice + ncpus: 12 + exe: /g/data/access/payu/access-esm/bin/coe/cicexx + input: + - /g/data/access/payu/access-esm/input/pre-industrial/ice
+ - name: coupler + model: oasis + ncpus: 0 + input: + - /g/data/access/payu/access-esm/input/pre-industrial/coupler + ``` + + {{ model }} is a coupled model deploying multiple submodels (i.e. [model components](/models/configurations/access-esm/#model-components)). + This section specifies the submodels and configuration options required to execute the model correctly.
+ Each submodel contains additional configuration options that are read in when the submodel is running. These options are specified in the subfolder of the _control_ directory, whose name matches the submodel's `name` (e.g., configuration options for the `atmosphere` submodel are in the `~/access-esm/esm-pre-industrial/atmosphere` directory). + +- **Collate**
+ ```yaml + collate: + exe: /g/data/access/payu/access-esm/bin/mppnccombine + restart: true + mem: 4GB + ``` + The `collate` process combines a number of smaller files, which contain different parts of the model grid, into target output files. Restart files are typically tiled in the same way and will also be combined together if the `restart` option is set to `true`. + +- **Restart**
+ ```yaml + restart: /g/data/access/payu/access-esm/restart/pre-industrial + ``` + This is the location of the files used for a warm restart. + +- **Start date and run length**{: id="runtime"}
+ ```yaml + calendar: + start: + year: 101 + month: 1 + days: 1 + runtime: + years: 1 + months: 0 + days: 0 + ``` + This section specifies the start date and run length. + + !!! warning + The _run length_ (controlled by [`runtime`](#runtime)) can be different from the _total experiment length_. Also, while `runtime` can be reduced, it should not be increased to more than 1 year to avoid errors. For more information about the difference between _run length_ and _total experiment length_, or how to run the model for more than 1 year, refer to the section [Run configuration for multiple years](#run-configuration-for-multiple-years). + +- **Number of runs per PBS submission**{: id="runspersub"}
+ ```yaml + runspersub: 1 + ``` + {{ model }} configurations are often run in multiple steps (or cycles), with _payu_ running a maximum of `runspersub` runs for every [PBS job] submission. + + !!! warning + If you increase `runspersub`, you may need to increase the `walltime` in the [PBS resources](#pbs-resources). + +To find out more about other configuration settings for the `config.yaml` file, check out [how to configure your experiment with _payu_](https://payu.readthedocs.io/en/latest/config.html). + +### Edit a single {{ model }} component configuration + +Each of [{{ model }} components](/models/configurations/access-esm/#model-components) contains additional configuration options that are read in when the model component is running.
+These options are typically useful to modify the physics used in the model, the input data, or the model variables saved in the output files. + +These configuration options are specified in files located inside a subfolder of the _control_ directory, named according to the submodel's `name` specified in the `config.yaml` `submodels` section (e.g., configuration options for the `atmosphere` submodel are in the `~/access-esm/esm-pre-industrial/atmosphere` directory).
+To modify these options please refer to the User Guide of the respective model component. + +## Run {{ model }} configuration + +After editing the configuration, you are ready to run {{ model }}.
+{{ model }} suites run on [_Gadi_](https://opus.nci.org.au/display/Help/0.+Welcome+to+Gadi#id-0.WelcometoGadi-Overview) through a [PBS job] submission managed by _payu_. + +### Payu setup (optional) + +As a first step, from within the _control_ directory, it is good practice to run: +``` +payu setup +``` + +This will prepare the model run, based on the experiment configuration. + + payu setup + laboratory path: /scratch/$PROJECT/$USER/access-esm + binary path: /scratch/$PROJECT/$USER/access-esm/bin + input path: /scratch/$PROJECT/$USER/access-esm/input + work path: /scratch/$PROJECT/$USER/access-esm/work + archive path: /scratch/$PROJECT/$USER/access-esm/archive + Loading input manifest: manifests/input.yaml + Loading restart manifest: manifests/restart.yaml + Loading exe manifest: manifests/exe.yaml + Setting up atmosphere + Setting up ocean + Setting up ice + Setting up coupler + Checking exe and input manifests + Updating full hashes for 3 files in manifests/exe.yaml + Creating restart manifest + Updating full hashes for 30 files in manifests/restart.yaml + Writing manifests/restart.yaml + Writing manifests/exe.yaml + + +!!! tip + This step can be skipped as it is also included in the run command. However, running it explicitly helps to check for errors and make sure executable and restart directories are accessible. + +### Run configuration + +To run {{ model }} configuration for one _run length_ (controlled by [`runtime`](#runtime) in the `config.yaml` file), from within the _control_ directory execute: + +``` +payu run -f +``` + +This will submit a single job to the queue with a _total experiment length_ of `runtime`. If there is no previous run, it will start from the [`start`](#runtime) date indicated in the `config.yaml` file. Otherwise, it will perform a warm restart from a previously saved restart file.
+ +!!! tip + The `-f` option ensures that _payu_ will run even if there is an existing non-empty `work` directory created from a previous failed run. + + + payu run -f + Loading input manifest: manifests/input.yaml + Loading restart manifest: manifests/restart.yaml + Loading exe manifest: manifests/exe.yaml + payu: Found modules in /opt/Modules/v4.3.0 + qsub -q normal -P <project> -l walltime=11400 -l ncpus=384 -l mem=1536GB -N pre-industrial -l wd -j n -v PAYU_PATH=/g/data/hh5/public/apps/miniconda3/envs/analysis3-23.01/bin,MODULESHOME=/opt/Modules/v4.3.0,MODULES_CMD=/opt/Modules/v4.3.0/libexec/modulecmd.tcl,MODULEPATH=/g/data3/hh5/public/modules:/etc/scl/modulefiles:/opt/Modules/modulefiles:/opt/Modules/v4.3.0/modulefiles:/apps/Modules/modulefiles -W umask=027 -l storage=gdata/access+gdata/hh5 -- /g/data/hh5/public/apps/miniconda3/envs/analysis3-23.01/bin/python3.9 /g/data/hh5/public/apps/miniconda3/envs/analysis3-23.01/bin/payu-run + <job-ID>.gadi-pbs + + +### Run configuration for multiple years + +If you want to run {{ model }} configuration for multiple _run lengths_ (controlled by [`runtime`](#runtime) in the `config.yaml` file), use the option `-n`: + +``` +payu run -f -n +``` + +This will run the configuration `number-of-runs` times with a _total experiment length_ of `runtime * number-of-runs`. The number of consecutive [PBS jobs][PBS job] submitted to the queue depends on the [`runspersub`](#runspersub) value specified in the config.yaml file. + +### Understand `runtime`, `runspersub`, and `-n` parameters + +With the correct use of [`runtime`](#runtime), [`runspersub`](#runspersub) and `-n` parameters, you can have full control of your run.
+ +- `runtime` defines the _run length_. +- `runspersub` defines the maximum number of runs for every [PBS job] submission. +- `-n` sets the number of runs to be performed. + +Now some practical examples: + +- **Run 20 years of simulation with resubmission every 5 years**
+ To have a _total experiment length_ of 20 years with a 5-year resubmission cycle, leave [`runtime`](#runtime) as the default value of `1 year` and set [`runspersub`](#runspersub) to `5`. Then, run the configuration with `-n` set to `20`: + ``` + payu run-f -n 20 + ``` + This will submit subsequent jobs for the following years: 1 to 5, 6 to 10, 11 to 15, and 16 to 20, which is a total of 4 PBS jobs. + +- **Run 7 years of simulation with resubmission every 3 years**
+ To have a _total experiment length_ of 7 years with a 3-year resubmission cycle, leave [`runtime`](#runtime) as the default value of `1 year` and set [`runspersub`](#runspersub) to `3`. Then, run the configuration with `-n` set to `7`: + ``` + payu run -f -n 7 + ``` + This will submit subsequent jobs for the following years: 1 to 3, 4 to 6, and 7, which is a total of 3 PBS jobs. + +- **Run 3 months and 10 days of simulation in a single submission**
+ To have a _total experiment length_ of 3 months and 10 days in a single submission, set the [`runtime`](#runtime) as follows: + ```yaml + years: 0 + months: 3 + days: 10 + ``` + Set [`runspersub`](#runspersub) to `1` (or any value > 1) and run the configuration without `-n` option (or with `-n` set to `1`): + ``` + payu run -f + ``` + +- **Run 1 year and 4 months of simulation with resubmission every 4 months**
+ To have a _total experiment length_ of 1 year and 4 months (16 months), you need to split it into multiple runs. For example, 4 runs of 4 months each. In this case, set the [`runtime`](#runtime) as follows: + ```yaml + years: 0 + months: 4 + days: 0 + ``` + Since the _run length_ is set to 4 months, set [`runspersub`](#runspersub) to `1` to resubmit your jobs every 4 months (i.e. every run). Then, run the configuration with `-n` set to `4`: + ``` + payu run -f -n 4 + ``` + +## Monitor {{ model }} runs + +You can execute the following command to show the status of all your submitted [PBS jobs][PBS job]: + +``` +qstat -u $USER +``` + + qstat -u $USER + Job id                Name             User             Time Use S Queue + --------------------- ---------------- ---------------- -------- - ----- + <job-ID>.gadi-pbs     pre-industrial   <$USER>            <time> R normal-exec + <job-ID-2>.gadi-pbs   <other-job-name> <$USER>            <time> R normal-exec + <job-ID-3>.gadi-pbs   <other-job-name> <$USER>            <time> R normal-exec + + +If you changed the `jobname` in the [PBS resources](#pbs-resources) of the _Master Configuration_ file, that will appear as your job's _Name_ instead of `pre-industrial`.
+ +_S_ indicates the status of your run, where: + +- _Q_ → Job waiting in the queue to start +- _R_ → Job running +- _E_ → Job ending +- _H_ → Job on hold + +If there are no jobs listed with your `jobname` (or if no job is listed), your run either successfully completed or was terminated due to an error.
+For more information, check [NCI documentation](https://opus.nci.org.au/display/Help/FAQ+1%3A+Why+My+Jobs+are+NOT+Running). + +### Stop a run + +If you want to manually terminate a run, you can do so by executing: +``` +qdel +``` + +### Error and output log files + +While the model is running, _payu_ saves the standard output and standard error in the respective `access.out` and `access.err` files in the _control_ directory. You can examine the contents of these files to check on the status of a run as it progresses.
+When the model completes its run, or if it crashes, the output and error log files are by default renamed as `jobname.o` and `jobname.e`, respectively. + +### Model Live Diagnostics + +ACCESS-NRI developed the [Model Live Diagnostics](/model_evaluation/model_diagnostics) framework to check, monitor, visualise, and evaluate model behaviour and progress of ACCESS models currently running on _Gadi_.
+For a complete documentation on how to use this framework, check the [Model Diagnostics documentation](https://med-live-diagnostics.readthedocs.io/en/latest/index.html). + +## {{ model }} outputs + +At the end of the model run, output files (and restart files) are moved from the `work` directory into the `archive` directory under `/scratch/$PROJECT/$USER/access-esm/archive/access-esm`, where they are further subdivided for each run. They are also symlinked in the _control_ directory to `~/access-esm/archive`.
+The naming format for a typical output folder is `outputXXX` and for a restart folder `restartXXX`, where _XXX_ is the run number starting from `000`. + +!!! tip + A run with a different {{ model }} configuration (different git branch) counts as a new run.
+ Thus, if output folders already exist, the internal number of the new output folder will be set to the first available _XXX_ number. + +Outputs and restarts are separated in the respective folders for each model component.
+For the atmospheric output data, the files are usually [UM fieldsfile](https://code.metoffice.gov.uk/doc/um/latest/papers/umdp_F03.pdf), formatted as `a.p`. + + cd /scratch/$PROJECT/$USER/access-esm/archive/esm-pre-industrial + ls + output000 pbs_logs restart000 + ls output000/atmosphere + aiihca.daa1210 aiihca.daa1810 aiihca.paa1apr aiihca.paa1jun aiihca.pea1apr aiihca.pea1jun aiihca.pga1apr aiihca.pga1jun atm.fort6.pe0 exstat ihist prefix.CNTLGEN UAFLDS_A aiihca.daa1310 aiihca.daa1910 aiihca.paa1aug aiihca.paa1mar aiihca.pea1aug aiihca.pea1mar aiihca.pga1aug aiihca.pga1mar cable.nml fort.57 INITHIS prefix.PRESM_A um_env.py aiihca.daa1410 aiihca.daa1a10 aiihca.paa1dec aiihca.paa1may aiihca.pea1dec aiihca.pea1may aiihca.pga1dec aiihca.pga1may CNTLALL ftxx input_atm.nml SIZES xhist aiihca.daa1510 aiihca.daa1b10 aiihca.paa1feb aiihca.paa1nov aiihca.pea1feb aiihca.pea1nov aiihca.pga1feb aiihca.pga1nov CONTCNTL ftxx.new namelists STASHC aiihca.daa1610 aiihca.daa1c10 aiihca.paa1jan aiihca.paa1oct aiihca.pea1jan aiihca.pea1oct aiihca.pga1jan aiihca.pga1oct debug.root.01 ftxx.vars nout.000000 thist aiihca.daa1710 aiihca.daa2110 aiihca.paa1jul aiihca.paa1sep aiihca.pea1jul aiihca.pea1sep aiihca.pga1jul aiihca.pga1sep errflag hnlist prefix.CNTLATM UAFILES_A + + + +- [https://github.com/coecms/esm-pre-industrial](https://github.com/coecms/esm-pre-industrial) +- [https://payu.readthedocs.io/en/latest/usage.html](https://payu.readthedocs.io/en/latest/usage.html) + diff --git a/docs/models/run-a-model/run-access-om.md b/docs/models/run-a-model/run-access-om.md index 07afea68f..d0dad0063 100644 --- a/docs/models/run-a-model/run-access-om.md +++ b/docs/models/run-a-model/run-access-om.md @@ -1,24 +1,25 @@ {% set model = "ACCESS-OM2" %} -{% set access_om2_configs = "https://github.com/ACCESS-NRI/access-om2-configs" %} +{% set github_configs = "https://github.com/ACCESS-NRI/access-om2-configs" %} [cosima]: https://cosima.org.au [PBS job]: https://opus.nci.org.au/display/Help/4.+PBS+Jobs [payu]: https://github.com/payu-org/payu -[model components]: https://access-hive.org.au/models/configurations/access-om/#model-components +[model components]: /models/configurations/access-om/#model-components +[model configurations]: /models/configurations/access-om/#access-om2 [gadi]: https://opus.nci.org.au/display/Help/0.+Welcome+to+Gadi#id-0.WelcometoGadi-Overview -[:fontawesome-brands-github:{: class="twemoji icon-before-text"} {{ model }} configurations](https://github.com/ACCESS-NRI/access-om2-configs){: class="text-card"} +[:fontawesome-brands-github:{: class="twemoji icon-before-text"} {{ model }} configurations]({{github_configs}}){: class="text-card"} # Run {{ model }} ## About -{{ model }} is an Ocean Sea-Ice model. More information is available in the [{{ model }} overview](https://access-hive.org.au/models/configurations/access-om/#access-om2). +{{ model }} is an Ocean Sea-Ice model. More information is available in the [{{ model }} overview][model configurations]. The instructions below outline how to run {{ model }} using ACCESS-NRI's software deployment pipeline, specifically designed to run on the [National Computating Infrastructure (NCI)](https://nci.org.au/about-us/who-we-are) supercomputer [_Gadi_][gadi]. -If you are unsure whether {{ model }} is the right choice for your experiment, take a look at the overview of [ACCESS Models](https://access-hive.org.au/models). +If you are unsure whether {{ model }} is the right choice for your experiment, take a look at the overview of [ACCESS Models](/models). -All Model configurations are open source, licensed under [CC BY 4.0](https://creativecommons.org/licenses/by/4.0/?ref=chooser-v1")![CC icon](https://mirrors.creativecommons.org/presskit/icons/cc.svg?ref=chooser-v1){: style="height:1em;margin-left:0.2em;vertical-align:text-top;"}![BY icon](https://mirrors.creativecommons.org/presskit/icons/by.svg?ref=chooser-v1){: style="height:1em;margin-left:0.2em;vertical-align:text-top;"} and available on [ACCESS-NRI GitHub](https://github.com/ACCESS-NRI/access-om2-configs). +All {{model}} configurations are open source, licensed under [CC BY 4.0](https://creativecommons.org/licenses/by/4.0/?ref=chooser-v1")![CC icon](https://mirrors.creativecommons.org/presskit/icons/cc.svg?ref=chooser-v1){: style="height:1em;margin-left:0.2em;vertical-align:text-top;"}![BY icon](https://mirrors.creativecommons.org/presskit/icons/by.svg?ref=chooser-v1){: style="height:1em;margin-left:0.2em;vertical-align:text-top;"} and available on [ACCESS-NRI GitHub]({{github_configs}}). {{ model }} release notes are [available on the ACCESS-Hive Forum](https://forum.access-hive.org.au/t/access-om2-release-information/1602/) and are updated when new releases are made available. @@ -26,7 +27,7 @@ All Model configurations are open source, licensed under [CC BY 4.0](https://cre ### General prerequisites -Before running {{ model }}, you need to fulfil general prerequisites outlined in the [First Steps](https://access-hive.org.au/getting_started/first_steps) section. +Before running {{ model }}, you need to fulfil general prerequisites outlined in the [First Steps](/getting_started/first_steps) section. ### Model-specific prerequisites @@ -57,12 +58,12 @@ Before running {{ model }}, you need to fulfil general prerequisites outlined in ## Get {{ model }} configuration -All released {{ model }} configurations are available from the [{{ model }} configs]({{ access_om2_configs }}) GitHub repository.
+All released {{ model }} configurations are available from the [{{ model }} configs]({{ github_configs }}) GitHub repository.
Released configurations are tested and supported by ACCESS-NRI, as an adaptation of those originally developed by [COSIMA][cosima]. -For more information on {{ model }} configurations, check [ACCESS-OM2](/models/configurations/access-om#access-om2) page. +For more information on {{ model }} configurations, check [{{model}}][model configurations] page. -More information about the available experiments and the naming scheme of the branches can also be found in the [{{ model }} configs]({{ access_om2_configs }}) GitHub repository. +More information about the available experiments and the naming scheme of the branches can also be found in the [{{ model }} configs]({{ github_configs }}) GitHub repository. The first step is to choose a configuration from those available.
For example, if the required configuration is the 1° horizontal resolution with repeat-year _JRA55_ forcing (without BGC), then the branch to select is [`release-1deg_jra55_ryf`](https://github.com/ACCESS-NRI/access-om2-configs/tree/release-1deg_jra55_ryf). @@ -71,7 +72,7 @@ To clone this branch to a location on _Gadi_, run: mkdir -p ~/access-om2 cd ~/access-om2 - payu clone -b expt -B release-1deg_jra55_ryf {{ access_om2_configs }} 1deg_jra55_ryf + payu clone -b expt -B release-1deg_jra55_ryf {{ github_configs }} 1deg_jra55_ryf In the example above the `payu clone` command clones the 1° repeat-year JRA55 configuration (` -B release-1deg_jra55_ryf`) to a new experiment branch (`-b expt`) to a directory named `1deg_jra55_ryf`. @@ -155,20 +156,6 @@ For information about `restart_period`, refer to [Change run length](#change-run !!! tip You can add the `-f` option to `payu run` to let the model run even if there is an existing non-empty `work` directory, created from a previous failed run or from running `payu setup`. -### Run configuration for more than 5 years - -As mentioned in the [Change run length](#change-run-length) section, you cannot specify more than 5 years as `restart_period`.
-If you want to run a configuration for more than 5 years, you need to use the `-n` option: - - payu run -n - -This will run {{ model }} `number-of-runs` consecutive times, each with a *run length* equal to `restart_period`.
-This way, the *total experiment length* will be `restart_period * number-of-runs`. - -For example, to run a configuration for a total of 50 years with a `restart_period` of 5 years, the `number-of-runs` should be set to `10`: - - payu run -n 10 - ---------------------------------------------------------------------------------------- ## Monitor {{ model }} runs @@ -339,7 +326,21 @@ For example, to make the model run for 1 year, 4 months and 10 days, change `res While `restart_period` can be reduced, it should not be increased to more than 5 years to avoid errors.

It is also important to differentiate between _run length_ and _total experiment length_.
- For more information about their difference, or how to run the model for more than 5 years, refer to the section [Run configuration for more than 5 years](#run-configuration-for-more-than-5-years). + For more information about their difference, or how to run the model for more than 5 years, refer to [Run configuration for more than 5 years](#run-configuration-for-more-than-5-years). + +#### Run configuration for more than 5 years + +As mentioned in the [Change run length](#change-run-length) section, you cannot specify more than 5 years as `restart_period`.
+If you want to run a configuration for more than 5 years, you need to use the `-n` option: + + payu run -n + +This will run {{ model }} `number-of-runs` consecutive times, each with a *run length* equal to `restart_period`.
+This way, the *total experiment length* will be `restart_period * number-of-runs`. + +For example, to run a configuration for a total of 50 years with a `restart_period` of 5 years, the `number-of-runs` should be set to `10`: + + payu run -n 10 ### Modify PBS resources From 72f10522231358e673a5a4d2bf72d35dc2e19de4 Mon Sep 17 00:00:00 2001 From: Davide Marchegiani Date: Tue, 16 Jul 2024 09:58:54 +1000 Subject: [PATCH 02/14] Rebased development --- docs/models/run-a-model/run-access-esm.md | 9 ++++++--- docs/models/run-a-model/run-access-om.md | 2 +- 2 files changed, 7 insertions(+), 4 deletions(-) diff --git a/docs/models/run-a-model/run-access-esm.md b/docs/models/run-a-model/run-access-esm.md index a7744d5f9..8d7d23033 100644 --- a/docs/models/run-a-model/run-access-esm.md +++ b/docs/models/run-a-model/run-access-esm.md @@ -129,7 +129,7 @@ _Payu_ also creates symbolic links in the _control_ directory pointing to the `a This design allows multiple self-resubmitting experiments that share common executables and input data to be run simultaneously. !!! admonition warning - The _laboratory_ directory might get deleted if not accessed for several days. To know more about how to preserve your experiment refer to [Syncing output data](#syncing-output-data). + Files on the `/scratch` drive, such as the _laboratory_ directory, might get deleted if not accessed for several days and the `/scratch` drive is limited in space. For these reasons, all model runs which are to be kept should be moved to `/g/data/` by enabling the _sync_ step in _payu_. To know more refer to [Syncing output data](#syncing-output-data). ### Run configuration @@ -378,7 +378,7 @@ project: ol01 ### Syncing output data The _laboratory_ directory is typically under the `/scratch` storage on _Gadi_, where [files are regularly deleted once they have been unaccessed for a period of time](https://opus.nci.org.au/pages/viewpage.action?pageId=156434436). For this reason climate model outputs need to be moved to a location with longer term storage.
-On _Gadi_, this is typically under a project code on `/g/data`. +On _Gadi_, this is typically in a folder under a project code on `/g/data`. _Payu_ has built-in support to sync outputs, restarts and a copy of the _control_ directory git history to another location.
This feature is controlled by the following section in the `config.yaml` file: @@ -392,7 +392,10 @@ sync: - '*.nc.*' - 'iceh.????-??-??.nc' ``` -To enable output syncing, change `enable` to `True`, and set `path` to a location on `/g/data`. +To enable syncing, change `enable` to `True`, and set `path` to a location on `/g/data`, where _payu_ will copy output and restart folders. A sensible `path` could be: `/g/data/$PROJECT/$USER/{{model}}/experiment_name/`. + +!!! admonition tip + The {{model}} default configurations include a [userscript](#userscripts) in the _sync_ step that concatenates daily history/diagnostic output from the Sea-Ice model (CICE5) into monthly files. This speeds up access and saves storage space, but will only run if _sync_ is enabled. ### Saving model restarts diff --git a/docs/models/run-a-model/run-access-om.md b/docs/models/run-a-model/run-access-om.md index d0dad0063..8d79cdf45 100644 --- a/docs/models/run-a-model/run-access-om.md +++ b/docs/models/run-a-model/run-access-om.md @@ -391,7 +391,7 @@ sync: - '*.nc.*' - 'iceh.????-??-??.nc' ``` -To enable syncing, change `enable` to `True`, and set `path` to a location on `/g/data`, where _payu_ will copy output and restart folders. A sensible `path` could be: `/g/data/$PROJECT/$USER/experiment_name/`. +To enable syncing, change `enable` to `True`, and set `path` to a location on `/g/data`, where _payu_ will copy output and restart folders. A sensible `path` could be: `/g/data/$PROJECT/$USER/{{model}}/experiment_name/`. !!! admonition tip The {{model}} default configurations include a [userscript](#userscripts) in the _sync_ step that concatenates daily history/diagnostic output from the Sea-Ice model (CICE5) into monthly files. This speeds up access and saves storage space, but will only run if _sync_ is enabled. From 10fa8d2709304813c21b9f9ce86ac7a3c6da6dd0 Mon Sep 17 00:00:00 2001 From: Spencer Wong Date: Wed, 14 Aug 2024 13:38:40 +1000 Subject: [PATCH 03/14] Add ESM info and TODOs --- docs/models/run-a-model/run-access-esm.md | 207 +++++++++++++--------- 1 file changed, 128 insertions(+), 79 deletions(-) diff --git a/docs/models/run-a-model/run-access-esm.md b/docs/models/run-a-model/run-access-esm.md index 8d7d23033..64fce86d5 100644 --- a/docs/models/run-a-model/run-access-esm.md +++ b/docs/models/run-a-model/run-access-esm.md @@ -52,7 +52,7 @@ Before running {{ model }}, you need to fulfil general prerequisites outlined in !!! warning _payu_ version >=1.1.3 is required - TODO: Check if any specific version is required + TODO: Update this with the next version number for payu when it is released, as ESM1.5 configs will require the latest changes. ---------------------------------------------------------------------------------------- @@ -66,7 +66,7 @@ For more information on {{ model }} configurations, check [{{model}}][model conf More information about the available experiments and the naming scheme of the branches can also be found in the [{{ model }} configs]({{ github_configs }}) GitHub repository. The first step is to choose a configuration from those available.
-TODO: check correct configuration until line 106 +TODO: check correct configuration until line 106 - replace with pre-industrial when ready For example, if the required configuration is the 1° horizontal resolution with repeat-year _JRA55_ forcing (without BGC), then the branch to select is [`release-1deg_jra55_ryf`](https://github.com/ACCESS-NRI/access-om2-configs/tree/release-1deg_jra55_ryf). To clone this branch to a location on _Gadi_, run: @@ -129,7 +129,7 @@ _Payu_ also creates symbolic links in the _control_ directory pointing to the `a This design allows multiple self-resubmitting experiments that share common executables and input data to be run simultaneously. !!! admonition warning - Files on the `/scratch` drive, such as the _laboratory_ directory, might get deleted if not accessed for several days and the `/scratch` drive is limited in space. For these reasons, all model runs which are to be kept should be moved to `/g/data/` by enabling the _sync_ step in _payu_. To know more refer to [Syncing output data](#syncing-output-data). + Files on the `/scratch` drive, such as the _laboratory_ directory, might get deleted if not accessed for several days and the `/scratch` drive is limited in space. For these reasons, all model runs which are to be kept should be moved to `/g/data/` by enabling the _sync_ step in _payu_. To know more refer to [Syncing output data](#syncing-output-data). TODO: find out whether sync will be available at the time of ESM1.5 release. ### Run configuration @@ -137,8 +137,9 @@ To run {{ model }} configuration execute the following command from within the * payu run -This will submit a single job to the queue with a run length of `restart_period`.
-For information about `restart_period`, refer to [Change run length](#change-run-length). +This will submit a single job to the queue with a run length given by [`runtime`](# TODO: ADD LINK) in the `config.yaml` file.
+ + cd ~/access-om2/1deg_jra55_ryf @@ -160,6 +161,8 @@ For information about `restart_period`, refer to [Change run length](#change-run ---------------------------------------------------------------------------------------- ## Monitor {{ model }} runs +TODO: REPLACE CONFIG NAMES WITH ESM CONFIG + The `payu run` command prints out the PBS `job-ID` (formatted as `<9-digit-number>.gadi-pbs`), as the last line to the terminal.
To print out information on the status of a specific job, you can execute the following command: @@ -248,6 +251,8 @@ This command will: - generate manifests - report useful information to the user, such as the location of the _laboratory_ where the `work` and `archive` directories are located +TODO: UPDATE PATHS FOR ESM + payu setup laboratory path: /scratch/$PROJECT/$USER/access-om2 @@ -285,6 +290,7 @@ Output and restart folders are called `outputXXX` and `restartXXX`, respectively Model components are separated into subdirectories within the output and restart directories. +TODO: UPDATE PATH FOR ESM1.5 cd ~/access-om2/1deg_jra55_ryf ls @@ -307,46 +313,93 @@ To find out more about configuration settings for the `config.yaml` file, refer One of the most common changes is to adjust the duration of the model run.
For example, when debugging changes to a model, it is common to reduce the run length to minimise resource consumption and return faster feedback on changes. -The run length is controlled by the `restart_period` field in the `&date_manager_nml` section of the `~/access-om2/1deg_jra55_ryf/accessom2.nml` file: +The length of an {{model}} run is controlled by the `runtime` settings in the `config.yaml` file: + +``` + runtime: + years: 1 + months: 0 + days: 0 +``` + +For example, to make the model run for a single month, change the `runtime` to +``` + runtime: + years: 0 + months: 1 + days: 0 +``` +!!! warning + The atmospheric component of {{ model }} is configured to produce restart files at the end of each month. To ensure that a valid restart file is produced at the end of a run, `runtime` setting for {{ model }} should total an integer number of months + +!!! warning + The _run length_ (controlled by [`runtime`](#runtime)) can be different from the _total experiment length_. While `runtime` can be reduced, it should not be increased to more than 1 year to avoid errors. For more information about the difference between _run length_ and _total experiment length_, or how to run the model for more than 1 year, refer to the section [Understand `runtime`, `runspersub`, and `-n` parameters](#run-configuration-for-multiple-years). TODO: UPDATE LINK' + +If you want to run {{ model }} configuration for multiple _run lengths_ (controlled by [`runtime`](#runtime) in the `config.yaml` file), use the option `-n`: +TODO: DOES THE RUNTIME LINK WORK? + +``` +payu run -f -n +``` - &date_manager_nml - forcing_start_date = '1958-01-01T00:00:00' - forcing_end_date = '2019-01-01T00:00:00'
- ! Runtime for a single segment/job/submit, format is years, months, seconds, - ! two of which must be zero. - restart_period = 5, 0, 0 - &end +This will run the configuration `number-of-runs` times with a _total experiment length_ of `runtime * number-of-runs`. The number of consecutive [PBS jobs][PBS job] submitted to the queue depends on the [`runspersub`](#runspersub) value specified in the config.yaml file. -The format is `restart_period = , , `. +### Understand `runtime`, `runspersub`, and `-n` parameters -For example, to make the model run for 1 year, 4 months and 10 days, change `restart_period` to: +With the correct use of [`runtime`](#runtime), [`runspersub`](#runspersub) and `-n` parameters, you can have full control of your run.
- restart_period = 1, 4, 10 +- `runtime` defines the _run length_. +- `runspersub` defines the maximum number of runs for every [PBS job] submission. +- `-n` sets the number of runs to be performed. -!!! warning - While `restart_period` can be reduced, it should not be increased to more than 5 years to avoid errors. -

- It is also important to differentiate between _run length_ and _total experiment length_.
- For more information about their difference, or how to run the model for more than 5 years, refer to [Run configuration for more than 5 years](#run-configuration-for-more-than-5-years). +Now some practical examples: -#### Run configuration for more than 5 years +- **Run 20 years of simulation with resubmission every 5 years**
+ To have a _total experiment length_ of 20 years with a 5-year resubmission cycle, leave [`runtime`](#runtime) as the default value of `1 year` and set [`runspersub`](#runspersub) to `5`. Then, run the configuration with `-n` set to `20`: + ``` + payu run-f -n 20 + ``` + This will submit subsequent jobs for the following years: 1 to 5, 6 to 10, 11 to 15, and 16 to 20, which is a total of 4 PBS jobs. -As mentioned in the [Change run length](#change-run-length) section, you cannot specify more than 5 years as `restart_period`.
-If you want to run a configuration for more than 5 years, you need to use the `-n` option: +- **Run 7 years of simulation with resubmission every 3 years**
+ To have a _total experiment length_ of 7 years with a 3-year resubmission cycle, leave [`runtime`](#runtime) as the default value of `1 year` and set [`runspersub`](#runspersub) to `3`. Then, run the configuration with `-n` set to `7`: + ``` + payu run -f -n 7 + ``` + This will submit subsequent jobs for the following years: 1 to 3, 4 to 6, and 7, which is a total of 3 PBS jobs. - payu run -n +- **Run 3 months and 10 days of simulation in a single submission**
+ To have a _total experiment length_ of 3 months and 10 days in a single submission, set the [`runtime`](#runtime) as follows: + ```yaml + years: 0 + months: 3 + days: 10 + ``` + Set [`runspersub`](#runspersub) to `1` (or any value > 1) and run the configuration without `-n` option (or with `-n` set to `1`): + ``` + payu run -f + ``` -This will run {{ model }} `number-of-runs` consecutive times, each with a *run length* equal to `restart_period`.
-This way, the *total experiment length* will be `restart_period * number-of-runs`. +- **Run 1 year and 4 months of simulation with resubmission every 4 months**
+ To have a _total experiment length_ of 1 year and 4 months (16 months), you need to split it into multiple runs. For example, 4 runs of 4 months each. In this case, set the [`runtime`](#runtime) as follows: + ```yaml + years: 0 + months: 4 + days: 0 + ``` + Since the _run length_ is set to 4 months, set [`runspersub`](#runspersub) to `1` to resubmit your jobs every 4 months (i.e. every run). Then, run the configuration with `-n` set to `4`: + ``` + payu run -f -n 4 + ``` -For example, to run a configuration for a total of 50 years with a `restart_period` of 5 years, the `number-of-runs` should be set to `10`: - payu run -n 10 ### Modify PBS resources If the model has been altered and needs more time to complete, more memory, or needs to be submitted under a different NCI project, you will need to modify the following section in the `config.yaml`: +TODO: update with esm config info + ```yaml # If submitting to a different project to your default, uncomment line below # and replace PROJECT_CODE with appropriate code. This may require setting shortpath @@ -361,6 +414,8 @@ jobname: 1deg_jra55_ryf mem: 1000GB ``` +TODO: Update with a relevant project + These lines can be edited to change the [PBS directives](https://opus.nci.org.au/display/Help/PBS+Directives+Explained) for the [PBS job][PBS job]. For example, to run {{ model }} under the `ol01` project (COSIMA Working Group), uncomment the line beginning with `# project` by deleting the `#` symbol and replace `PROJECT_CODE` wih `ol01`: @@ -376,6 +431,7 @@ project: ol01 To run {{ model }}, you need to be a member of a project with allocated _Service Units (SU)_. For more information, check [how to join relevant NCI projects](/getting_started/first_steps#join-relevant-nci-projects). ### Syncing output data +TODO: Wait for decisions/updates on ESM1.5 syncing. If not part of release, delete this section. The _laboratory_ directory is typically under the `/scratch` storage on _Gadi_, where [files are regularly deleted once they have been unaccessed for a period of time](https://opus.nci.org.au/pages/viewpage.action?pageId=156434436). For this reason climate model outputs need to be moved to a location with longer term storage.
On _Gadi_, this is typically in a folder under a project code on `/g/data`. @@ -399,6 +455,7 @@ To enable syncing, change `enable` to `True`, and set `path` to a location on `/ ### Saving model restarts +TODO: Wait for finalisation of date based restarts for ESM {{ model }} outputs restart files after every run to allow for subsequent runs to start from a previously saved model state.
Restart files can occupy a significant amount of disk space, and keeping a lot of them is often not necessary. @@ -428,14 +485,12 @@ For more information, check [_payu_ Configuration Settings documentation](https: #### Model configuration -This section tells _payu_ which driver to use for the main model configuration (`access-om2`) and the location of all inputs that are common to all its [model components]. +This section tells _payu_ which driver to use for the main model (`access` refers to {{ model }}). ```yaml -name: common -model: access-om2 -input: /g/data/ik11/inputs/access-om2/input_20201102/common_1deg_jra55 +model: access ``` -The `name` field is not actually used for the configuration run, so it can be safely ignored. + #### Submodels @@ -443,57 +498,48 @@ The `name` field is not actually used for the configuration run, so it can be sa This section specifies the submodels and configuration options required to execute {{ model }} correctly. -Each submodel contains additional configuration options that are read in when the submodel is running. These options are specified in the subfolder of the _control_ directory whose name matches the submodel's `name` (e.g., configuration options for the `ocean` submodel are in the `~/access-om2/1deg_jra55_ryf/ocean` directory). +Each submodel contains additional configuration options that are read in when the submodel is running. These options are specified in the subfolder of the _control_ directory, whose name matches the submodel's `name` (e.g., configuration options for the `atmosphere` submodel are in the `~/access-esm/esm-pre-industrial/atmosphere` directory). + +#TODO: UPDATE WITH UPDATED ESM1.5 CONFIG FILE. ??? code "Expand to show the full `submodels` section" ```yaml submodels: - - name: atmosphere - model: yatm - exe: /g/data/vk83/apps/spack/0.20/release/linux-rocky8-x86_64/intel-19.0.5.281/libaccessom2-git.2023.10.26=2023.10.26-ieiy3e7hidn4dzaqly3ly2yu45mecgq4/bin/yatm.exe - input: - - /g/data/vk83/experiments/inputs/access-om2/remapping_weights/JRA55/global.1deg/2020.05.30/rmp_jrar_to_cict_CONSERV.nc - - /g/data/vk83/experiments/inputs/JRA-55/RYF/v1-4/data - ncpus: 1 - - - name: ocean - model: mom - exe: /g/data/vk83/apps/spack/0.20/release/linux-rocky8-x86_64/intel-19.0.5.281/mom5-git.2023.11.09=2023.11.09-ewcdbrfukblyjxpkhd3mfkj4yxfolal4/bin/fms_ACCESS-OM.x - input: - - /g/data/vk83/experiments/inputs/access-om2/ocean/grids/mosaic/global.1deg/2020.05.30/grid_spec.nc - - /g/data/vk83/experiments/inputs/access-om2/ocean/grids/mosaic/global.1deg/2020.05.30/ocean_hgrid.nc - - /g/data/vk83/experiments/inputs/access-om2/ocean/grids/mosaic/global.1deg/2020.05.30/ocean_mosaic.nc - - /g/data/vk83/experiments/inputs/access-om2/ocean/grids/bathymetry/global.1deg/2020.10.22/topog.nc - - /g/data/vk83/experiments/inputs/access-om2/ocean/grids/bathymetry/global.1deg/2020.10.22/ocean_mask.nc - - /g/data/vk83/experiments/inputs/access-om2/ocean/grids/vertical/global.1deg/2020.10.22/ocean_vgrid.nc - - /g/data/vk83/experiments/inputs/access-om2/ocean/processor_masks/global.1deg/216.16x15/2020.05.30/ocean_mask_table - - /g/data/vk83/experiments/inputs/access-om2/ocean/chlorophyll/global.1deg/2020.05.30/chl.nc - - /g/data/vk83/experiments/inputs/access-om2/ocean/initial_conditions/global.1deg/2020.10.22/ocean_temp_salt.res.nc - - /g/data/vk83/experiments/inputs/access-om2/ocean/tides/global.1deg/2020.05.30/tideamp.nc - - /g/data/vk83/experiments/inputs/access-om2/ocean/tides/global.1deg/2020.05.30/roughness_amp.nc - - /g/data/vk83/experiments/inputs/access-om2/ocean/tides/global.1deg/2020.05.30/roughness_cdbot.nc - - /g/data/vk83/experiments/inputs/access-om2/ocean/surface_salt_restoring/global.1deg/2020.05.30/salt_sfc_restore.nc - ncpus: 216 - - - name: ice - model: cice5 - exe: /g/data/vk83/apps/spack/0.20/release/linux-rocky8-x86_64/intel-19.0.5.281/cice5-git.2023.10.19=2023.10.19-rh3xfkrgajya3ghtliacuhlx3pgvrzqs/bin/cice_auscom_360x300_24x1_24p.exe - input: - - /g/data/vk83/experiments/inputs/access-om2/ice/grids/global.1deg/2020.05.30/grid.nc - - /g/data/vk83/experiments/inputs/access-om2/ice/grids/global.1deg/2020.10.22/kmt.nc - - /g/data/vk83/experiments/inputs/access-om2/ice/initial_conditions/global.1deg/2020.05.30/i2o.nc - - /g/data/vk83/experiments/inputs/access-om2/ice/initial_conditions/global.1deg/2020.05.30/o2i.nc - - /g/data/vk83/experiments/inputs/access-om2/ice/initial_conditions/global.1deg/2020.05.30/u_star.nc - - /g/data/vk83/experiments/inputs/access-om2/ice/initial_conditions/global.1deg/2020.05.30/monthly_sstsss.nc - ncpus: 24 + - name: atmosphere + model: um + ncpus: 192 + exe: /g/data/access/payu/access-esm/bin/coe/um7.3x + input: + - /g/data/access/payu/access-esm/input/pre-industrial/atmosphere + - /g/data/access/payu/access-esm/input/pre-industrial/start_dump
+ - name: ocean + model: mom + ncpus: 180 + exe: /g/data/access/payu/access-esm/bin/coe/mom5xx + input: + - /g/data/access/payu/access-esm/input/pre-industrial/ocean/common + - /g/data/access/payu/access-esm/input/pre-industrial/ocean/pre-industrial
+ - name: ice + model: cice + ncpus: 12 + exe: /g/data/access/payu/access-esm/bin/coe/cicexx + input: + - /g/data/access/payu/access-esm/input/pre-industrial/ice
+ - name: coupler + model: oasis + ncpus: 0 + input: + - /g/data/access/payu/access-esm/input/pre-industrial/coupler ``` #### Collate -Rather than outputting a single diagnostic file over the whole model horizontal grid, [MOM](/models/model_components/ocean/#modular-ocean-model-mom) typically generates diagnostic outputs as tiles, each of which spans over a portion of the model horizontal grid. +Rather than outputting a single diagnostic file over the whole model horizontal grid, the ocean component [MOM](/models/model_components/ocean/#modular-ocean-model-mom) typically generates diagnostic outputs as tiles, each of which spans over a portion of the model horizontal grid. -The `collate` section in the `yaml.conf` file controls the process that combines these smaller files into a single output file. +The `collate` section in the `config.yaml` file controls the process that combines these smaller files into a single output file. + +TODO: UPDATE WITH CORRECT DATA FOR AN ESM1.5 CONFIG ```yaml collate: restart: true @@ -528,7 +574,7 @@ In the example above, the default number of cpus per node is set to 48. This might need changing if the configuration is run on hardware with different node structure. #### Userscripts - +TODO: UPDATE WITH RELEVANT ESM1.5 USERSCRIPTS... WE PROBABLY WON'T BE CALLING ANY IN THE PI RUN... ```yaml userscripts: error: tools/resub.sh @@ -544,16 +590,17 @@ A dictionary to run scripts or subcommands at various stages of a _payu_ submiss For more information about specific `userscripts` fields, check the relevant section of [_payu_ Configuration Settings documentation](https://payu.readthedocs.io/en/latest/config.html#postprocessing). + +### Postscripts +TODO: ADD POSTSCRIPT EXAMPLE ONCE WE MOVE CONVERSION OVER THERE. + #### Miscellaneous The following configuration settings should never require changing: ```yaml stacksize: unlimited -mpirun: --mca io ompio --mca io_ompio_num_aggregators 1 qsub_flags: -W umask=027 -env: - UCX_LOG_LEVEL: 'error' ``` ### Edit a single {{ model }} component configuration @@ -561,6 +608,8 @@ env: Each of [{{ model }} components][model components] contains additional configuration options that are read in when the model component is running.
These options are typically useful to modify the physics used in the model, the input data, or the model variables saved in the output files. +TODO: UPDATE WITH CORRECT ESM PATH ONCE NAMING SCHEME CHOSEN + These configuration options are specified in files located inside a subfolder of the _control_ directory, named according to the submodel's `name` specified in the `config.yaml` `submodels` section (e.g., configuration options for the _ocean_ component are in the `~/access-om2/1deg_jra55_ryf/ocean` directory).
To modify these options please refer to the User Guide of the respective model component. From ba0cea5533b93b228b7314556476e9789822954f Mon Sep 17 00:00:00 2001 From: Spencer Wong Date: Wed, 14 Aug 2024 16:27:57 +1000 Subject: [PATCH 04/14] Add ESM specific information --- docs/models/configurations/access-esm.md | 10 + docs/models/run-a-model/run-access-esm.md | 331 +++++++++++----------- 2 files changed, 179 insertions(+), 162 deletions(-) diff --git a/docs/models/configurations/access-esm.md b/docs/models/configurations/access-esm.md index fbcc74dc5..8034e5ee4 100644 --- a/docs/models/configurations/access-esm.md +++ b/docs/models/configurations/access-esm.md @@ -9,6 +9,16 @@ This means it can simulate both the physical climate and global biogeochemical c [ACCESS-ESM1.5](https://www.publish.csiro.au/es/ES19035) [@Ziehn2020] is a fully-coupled climate model with land and ocean carbon cycle components. ACCESS-ESM1.5 was developed primarily to enable Australia to participate in the [Coupled Model Intercomparison Project Phase 6 (CMIP6)](https://wcrp-cmip.org/cmip6/) with an Earth System Model (ESM) version. +ACCESS-NRI has released ACCESS-ESM1.5 configurations as an adaptation of those originally developed by [CSIRO](https://www.csiro.au/en/research/environmental-impacts/climate-change/climate-science-centre) and [CLEX CMS](https://github.com/coecms/access-esm). + +There are currently two supported configurations: + +- ***Pre-industrial concentration driven***:
+ A global coupled model configuration running in co2 concentration driven mode under pre-industrial forcings, as described in [Ziehn et al. (2020)](https://doi.org/10.1071/ES19035). + Pre-industrial forcing data including atmospheric co2 concentrations are primarily sourced from UKMO versions of CMIP6 inputs, with additional atmospheric forcings sourced from CMIP5 and land cover data adapted from [Lawrence et al. (2012)](https://doi.org/10.1175/JCLI-D-11-00256.1). +- ***Historical concentration driven***:
+ A global coupled model configuration running in co2 concentration driven mode under time varying historical (1850-2014) forcings, as described in [Ziehn et al. (2020)](https://doi.org/10.1071/ES19035). + Historical forcing data including atmospheric co2 concentrations are primarily sourced from UKMO versions of CMIP6 inputs, with land use change data adapted from the Land-Use Harmonisation 2 (LUH2) dataset developed for CMIP6 [(Hurtt et al. 2017)](https://doi.org/10.22033/ESGF/input4MIPs.1127). ### Model components - **Atmoshpere**: [UM7.3](/models/model_components/atmosphere#unified-model-um), GA7.1 science configuration.
diff --git a/docs/models/run-a-model/run-access-esm.md b/docs/models/run-a-model/run-access-esm.md index 64fce86d5..0ba18deb4 100644 --- a/docs/models/run-a-model/run-access-esm.md +++ b/docs/models/run-a-model/run-access-esm.md @@ -12,7 +12,7 @@ ## About -{{ model }} is a fully-coupled global climate model. More information is available in the [{{ model }} overview][model configurations]. +{{ model }} is a fully-coupled global climate model, combining atmoshpere, land, ocean, sea ice, ocean biogeochemistry and land biogeochemistry components. A description of the model and its components is available in the [{{ model }} overview][model configurations]. The instructions below outline how to run {{ model }} using ACCESS-NRI's software deployment pipeline, specifically designed to run on the [National Computating Infrastructure (NCI)](https://nci.org.au/about-us/who-we-are) supercomputer [_Gadi_][gadi]. @@ -20,7 +20,8 @@ If you are unsure whether {{ model }} is the right choice for your experiment, t All {{model}} configurations are open source, licensed under [CC BY 4.0](https://creativecommons.org/licenses/by/4.0/?ref=chooser-v1")![CC icon](https://mirrors.creativecommons.org/presskit/icons/cc.svg?ref=chooser-v1){: style="height:1em;margin-left:0.2em;vertical-align:text-top;"}![BY icon](https://mirrors.creativecommons.org/presskit/icons/by.svg?ref=chooser-v1){: style="height:1em;margin-left:0.2em;vertical-align:text-top;"} and available on [ACCESS-NRI GitHub]({{github_configs}}). -{{ model }} release notes are [available on the ACCESS-Hive Forum **TODO change with correct link**](https://forum.access-hive.org.au/t/access-om2-release-information/1602/) and are updated when new releases are made available. + +{{ model }} release notes are [available on the ACCESS-Hive Forum ](MISSING LINK) and are updated when new releases are made available. ## Prerequisites @@ -35,7 +36,7 @@ Before running {{ model }}, you need to fulfil general prerequisites outlined in For more information on joining specific NCI projects, refer to [How to connect to a project](https://opus.nci.org.au/display/Help/How+to+connect+to+a+project). - **Payu**
- [Payu][payu] is a workflow management tool for running numerical models in supercomputing environments, for which there is extensive [documentation](https://payu.readthedocs.io/en/latest/).
+ [_Payu_][payu] is a workflow management tool for running numerical models in supercomputing environments, for which there is extensive [documentation](https://payu.readthedocs.io/en/latest/).
_Payu_ on _Gadi_ is available through a dedicated `conda` environment in the _vk83_ project.
After joining the _vk83_ project, load the `payu` module: @@ -50,51 +51,51 @@ Before running {{ model }}, you need to fulfil general prerequisites outlined in payu --version 1.1.3 + !!! warning _payu_ version >=1.1.3 is required - TODO: Update this with the next version number for payu when it is released, as ESM1.5 configs will require the latest changes. ---------------------------------------------------------------------------------------- ## Get {{ model }} configuration All released {{ model }} configurations are available from the [{{ model }} configs]({{ github_configs }}) GitHub repository.
-Released configurations are tested and supported by ACCESS-NRI, as an adaptation of those originally developed by [CLEX CMS](https://github.com/coecms/access-esm). +Released configurations are tested and supported by ACCESS-NRI, as an adaptation of those originally developed by [CSIRO](https://www.csiro.au/en/research/environmental-impacts/climate-change/climate-science-centre) and [CLEX CMS](https://github.com/coecms/access-esm). For more information on {{ model }} configurations, check [{{model}}][model configurations] page. More information about the available experiments and the naming scheme of the branches can also be found in the [{{ model }} configs]({{ github_configs }}) GitHub repository. The first step is to choose a configuration from those available.
-TODO: check correct configuration until line 106 - replace with pre-industrial when ready -For example, if the required configuration is the 1° horizontal resolution with repeat-year _JRA55_ forcing (without BGC), then the branch to select is [`release-1deg_jra55_ryf`](https://github.com/ACCESS-NRI/access-om2-configs/tree/release-1deg_jra55_ryf). + +For example, if the required configuration is the co2 concentration driven pre-industrial configuration, then the branch to select is [`release-preindustrial+concentrations`](MISSING LINK). To clone this branch to a location on _Gadi_, run: mkdir -p ~/access-esm1.5 cd ~/access-esm1.5 - payu clone -b expt -B release-1deg_jra55_ryf {{ github_configs }} 1deg_jra55_ryf + payu clone -b expt -B release-preindustrial+concentrations {{ github_configs }} preindustrial+concentrations -In the example above the `payu clone` command clones the 1° repeat-year JRA55 configuration (` -B release-1deg_jra55_ryf`) -to a new experiment branch (`-b expt`) to a directory named `1deg_jra55_ryf`. +In the example above the `payu clone` command clones the concentration driven pre-industrial configuration (` -B -B release-preindustrial+concentrations`) +to a new experiment branch (`-b expt`) to a directory named `preindustrial+concentrations`. !!! admonition tip Anyone using a configuration is advised to clone only a single branch (as shown in the example above) and not the entire repository. - mkdir -p ~/access-om2 - cd ~/access-om2 - payu clone -b expt -B release-1deg_jra55_ryf https://github.com/ACCESS-NRI/access-om2-configs.git 1deg_jra55_ryf - Cloned repository from https://github.com/ACCESS-NRI/access-om2-configs.git to directory: .../access-om/1deg_jra55_ryf + mkdir -p ~/access-esm1.5 + cd ~/access-esm1.5 + payu clone -b expt -B release-preindustrial+concentrations {{ github_configs }} preindustrial+concentrations + Cloned repository from {{ github_configs }} to directory: .../access-esm1.5/preindustrial+concentrations Created and checked out new branch: expt - laboratory path: /scratch/.../access-om2 - binary path: /scratch/.../access-om2/bin - input path: /scratch/.../access-om2/input - work path: /scratch/.../access-om2/work - archive path: /scratch/.../access-om2/archive - Updated metadata. Experiment UUID: daeee7ff-07e4-4f93-823b-cb7c6e4bdb6e - Added archive symlink to /scratch/.../access-om2/archive/1deg_jra55_ryf-expt-daeee7ff + laboratory path: /scratch/.../access-esm + binary path: /scratch/.../access-esm/bin + input path: /scratch/.../access-esm/input + work path: /scratch/.../access-esm/work + archive path: /scratch/.../access-esm/archive + Updated metadata. Experiment UUID: 0635396b-678d-45f9-b81e-abd25a2d7bf0 + Added archive symlink to /scratch/.../access-esm/archive/preindustrial+concentrations-expt-0635396b To change directory to control directory run: - cd 1deg_jra55_ryf + cd preindustrial+concentrations !!! tip @@ -105,15 +106,14 @@ to a new experiment branch (`-b expt`) to a directory named `1deg_jra55_ryf`. ---------------------------------------------------------------------------------------- ## Run {{ model }} configuration -TODO: modify file paths for ACCESS-ESM If you want to modify your configuration, refer to [Edit {{ model }} configuration](#edit-{{ model.lower() }}-configuration). {{ model }} configurations run on [_Gadi_][gadi] through a [PBS job][PBS job] submission managed by [_payu_][payu]. The general layout of a _payu_-supported model run consists of two main directories: -- The _control_ directory contains the model configuration and serves as the execution directory for running the model (in this example, the cloned directory `~/access-om2/1deg_jra55_ryf`). -- The _laboratory_ directory, where all the model components reside. For {{ model }}, it is typically `/scratch/$PROJECT/$USER/access-om2`. +- The _control_ directory contains the model configuration and serves as the execution directory for running the model (in this example, the cloned directory `~/access-esm1.5/preindustrial+concentrations`). +- The _laboratory_ directory, where all the model components reside. For {{ model }}, it is typically `/scratch/$PROJECT/$USER/access-esm`. This separates the small text configuration files from the larger binary outputs and inputs. In this way, the _control_ directory can be in the `$HOME` directory (as it is the only filesystem actively backed-up on _Gadi_). The quotas for `$HOME` are low and strict, which limits what can be stored there, so it is not suitable for larger files. @@ -129,7 +129,7 @@ _Payu_ also creates symbolic links in the _control_ directory pointing to the `a This design allows multiple self-resubmitting experiments that share common executables and input data to be run simultaneously. !!! admonition warning - Files on the `/scratch` drive, such as the _laboratory_ directory, might get deleted if not accessed for several days and the `/scratch` drive is limited in space. For these reasons, all model runs which are to be kept should be moved to `/g/data/` by enabling the _sync_ step in _payu_. To know more refer to [Syncing output data](#syncing-output-data). TODO: find out whether sync will be available at the time of ESM1.5 release. + Files on the `/scratch` drive, such as the _laboratory_ directory, might get deleted if not accessed for several days and the `/scratch` drive is limited in space. For these reasons, all model runs which are to be kept should be moved to `/g/data/` by enabling the _sync_ step in _payu_. To know more refer to [Syncing output data](#syncing-output-data). ### Run configuration @@ -137,21 +137,17 @@ To run {{ model }} configuration execute the following command from within the * payu run -This will submit a single job to the queue with a run length given by [`runtime`](# TODO: ADD LINK) in the `config.yaml` file.
- - +This will submit a single job to the queue with a run length given by [`runtime`](#runtime) in the `config.yaml` file.
- cd ~/access-om2/1deg_jra55_ryf - payu run - payu: warning: Job request includes 47 unused CPUs. - payu: warning: CPU request increased from 241 to 288 + cd ~/access-esm1.5/preindustrial+concentrations + payu run Loading input manifest: manifests/input.yaml Loading restart manifest: manifests/restart.yaml Loading exe manifest: manifests/exe.yaml payu: Found modules in /opt/Modules/v4.3.0 - qsub -q normal -P tm70 -l walltime=10800 -l ncpus=288 -l mem=1000GB -N 1deg_jra55_ryf -l wd -j n -v PAYU_PATH=/g/data/hh5/public/apps/miniconda3/envs/analysis3-23.10/bin,MODULESHOME=/opt/Modules/v4.3.0,MODULES_CMD=/opt/Modules/v4.3.0/libexec/modulecmd.tcl,MODULEPATH=/g/data/hr22/modulefiles:/g/data/hh5/public/modules:/etc/scl/modulefiles:/opt/Modules/modulefiles:/opt/Modules/v4.3.0/modulefiles:/apps/Modules/modulefiles -W umask=027 -l storage=gdata/hh5+gdata/vk83 -- /g/data/hh5/public/apps/miniconda3/envs/analysis3-23.10/bin/python3.10 /g/data/hh5/public/apps/miniconda3/envs/analysis3-23.10/bin/payu-run + qsub -q normal -P tm70 -l walltime=9000 -l ncpus=384 -l mem=1536GB -N pre-industrial -l wd -j n -v PAYU_PATH=/g/data/vk83/apps/payu/1.1.4/bin,MODULESHOME=/opt/Modules/v4.3.0,MODULES_CMD=/opt/Modules/v4.3.0/libexec/modulecmd.tcl,MODULEPATH=/g/data/vk83/modules:/etc/scl/modulefiles:/apps/Modules/restricted-modulefiles/matlab_monash:/opt/Modules/modulefiles:/opt/Modules/v4.3.0/modulefiles:/apps/Modules/modulefiles -W umask=027 -l storage=gdata/vk83 -- /g/data/vk83/apps/payu/1.1.4/bin/python3.10 /g/data/vk83/apps/payu/1.1.4/bin/payu-run <job-ID>.gadi-pbs> @@ -161,9 +157,6 @@ This will submit a single job to the queue with a run length given by [`runtime` ---------------------------------------------------------------------------------------- ## Monitor {{ model }} runs -TODO: REPLACE CONFIG NAMES WITH ESM CONFIG - - The `payu run` command prints out the PBS `job-ID` (formatted as `<9-digit-number>.gadi-pbs`), as the last line to the terminal.
To print out information on the status of a specific job, you can execute the following command: ``` @@ -173,7 +166,7 @@ qstat qstat <job-ID> Job id                Name             User             Time Use S Queue --------------------- ---------------- ---------------- -------- - ----- - <job-ID>.gadi-pbs  1deg_jra55_ryf   <$USER>            <time> R normal-exec + <job-ID>.gadi-pbs  pre-industrial   <$USER>            <time> R normal-exec To show the status of all your submitted [PBS jobs][PBS job], you can execute the following command: @@ -185,13 +178,13 @@ qstat -u $USER qstat -u $USER Job id                Name             User             Time Use S Queue --------------------- ---------------- ---------------- -------- - ----- - <job-ID>.gadi-pbs   1deg_jra55_ryf   <$USER>            <time> R normal-exec + <job-ID>.gadi-pbs   pre-industrial   <$USER>            <time> R normal-exec <job-ID>.gadi-pbs   <other-job-name> <$USER>            <time> R normal-exec <job-ID>.gadi-pbs   <other-job-name> <$USER>            <time> R normal-exec -The default name of your job is the name of the _payu_ _control_ directory (`1deg_jra55_ryf` in the example above).
-This can be changed by altering the `jobname` in the [PBS resources section](#modify-pbs-resources) of the `config.yaml` file. +The default name of your job is the name of the _payu_ _control_ directory (`preindustrial+concentrations` in the example above).
+This can be overwritten by altering the `jobname` in the [PBS resources section](#modify-pbs-resources) of the `config.yaml` file. _S_ indicates the status of your run, where: @@ -228,7 +221,7 @@ payu sweep #### Model log files -While the model is running, _payu_ saves the model standard output and error streams in the `access-om2.out` and `access-om2.err` files inside the _control_ directory, respectively.
+While the model is running, _payu_ saves the model standard output and error streams in the `access.out` and `access.err` files inside the _control_ directory, respectively.
You can examine the contents of these files to check on the status of a run as it progresses (or after a failed run has completed). !!! warning @@ -251,27 +244,30 @@ This command will: - generate manifests - report useful information to the user, such as the location of the _laboratory_ where the `work` and `archive` directories are located -TODO: UPDATE PATHS FOR ESM payu setup - laboratory path: /scratch/$PROJECT/$USER/access-om2 - binary path: /scratch/$PROJECT/$USER/access-om2/bin - input path: /scratch/$PROJECT/$USER/access-om2/input - work path: /scratch/$PROJECT/$USER/access-om2/work - archive path: /scratch/$PROJECT/$USER/access-om2/archive + laboratory path: /scratch/$PROJECT/$USER/access-esm + binary path: /scratch/$PROJECT/$USER/access-esm/bin + input path: /scratch/$PROJECT/$USER/access-esm/input + work path: /scratch/$PROJECT/$USER/access-esm/work + archive path: /scratch/$PROJECT/$USER/access-esm/archive + Found experiment archive: /scratch/$PROJECT/$USER/access-esm/archive/preindustrial+concentrations-expt-0635396b + payu: Found modules in /opt/Modules/v4.3.0 + Loading access-esm1p5/2024.05.0 + Loading requirement: cice4/2024.05.21 mom5/access-esm1.5_2024.06.20 um7/2024.07.03 Loading input manifest: manifests/input.yaml Loading restart manifest: manifests/restart.yaml Loading exe manifest: manifests/exe.yaml + Making exe links Setting up atmosphere Setting up ocean Setting up ice - Setting up access-om2 + Setting up coupler Checking exe and input manifests - Updating full hashes for 3 files in manifests/exe.yaml Creating restart manifest + Updating full hashes for 29 files in manifests/restart.yaml Writing manifests/restart.yaml - Writing manifests/exe.yaml This can help to isolate issues such as permissions problems accessing files and directories, missing files or malformed/incorrect paths. @@ -290,11 +286,10 @@ Output and restart folders are called `outputXXX` and `restartXXX`, respectively Model components are separated into subdirectories within the output and restart directories. -TODO: UPDATE PATH FOR ESM1.5 - cd ~/access-om2/1deg_jra55_ryf - ls - output000 pbs_logs restart000 + cd ~/access-esm/preindustrial+concentrations/archive + ls + metadata.yaml output000 pbs_logs restart000 ---------------------------------------------------------------------------------------- @@ -308,7 +303,7 @@ The `config.yaml` file located in the _control_ directory is the _Master Configu To find out more about configuration settings for the `config.yaml` file, refer to [how to configure your experiment with payu](https://payu.readthedocs.io/en/latest/config.html). -### Change run length +### Change run length {: id="runtime"} One of the most common changes is to adjust the duration of the model run.
For example, when debugging changes to a model, it is common to reduce the run length to minimise resource consumption and return faster feedback on changes. @@ -330,13 +325,13 @@ For example, to make the model run for a single month, change the `runtime` to days: 0 ``` !!! warning - The atmospheric component of {{ model }} is configured to produce restart files at the end of each month. To ensure that a valid restart file is produced at the end of a run, `runtime` setting for {{ model }} should total an integer number of months + The sea ice component of {{ model }} is configured to produce restart files only at the end of each year. While simulations shorter than a year can be helpful for troubleshooting, they will not produce valid restart files and it will not be possible to continue them after the first run. If running more than one segment is required, the `runtime` setting for {{ model }} should be left at one year. !!! warning - The _run length_ (controlled by [`runtime`](#runtime)) can be different from the _total experiment length_. While `runtime` can be reduced, it should not be increased to more than 1 year to avoid errors. For more information about the difference between _run length_ and _total experiment length_, or how to run the model for more than 1 year, refer to the section [Understand `runtime`, `runspersub`, and `-n` parameters](#run-configuration-for-multiple-years). TODO: UPDATE LINK' + The _run length_ (controlled by `runtime`) can be different from the _total experiment length_. While `runtime` can be reduced, it should not be increased to more than 1 year to avoid errors. For more information about the difference between _run length_ and _total experiment length_, or how to run the model for more than 1 year, refer to the section [Understand `runtime`, `runspersub`, and `-n` parameters](#multiple-runs). + +If you want to run {{ model }} configuration for multiple _run lengths_ (each with duration [`runtime`](#runtime) in the `config.yaml` file), use the option `-n`: -If you want to run {{ model }} configuration for multiple _run lengths_ (controlled by [`runtime`](#runtime) in the `config.yaml` file), use the option `-n`: -TODO: DOES THE RUNTIME LINK WORK? ``` payu run -f -n @@ -344,9 +339,9 @@ payu run -f -n This will run the configuration `number-of-runs` times with a _total experiment length_ of `runtime * number-of-runs`. The number of consecutive [PBS jobs][PBS job] submitted to the queue depends on the [`runspersub`](#runspersub) value specified in the config.yaml file. -### Understand `runtime`, `runspersub`, and `-n` parameters +### Understand `runtime`, `runspersub`, and `-n` parameters {: id="multiple-runs"} -With the correct use of [`runtime`](#runtime), [`runspersub`](#runspersub) and `-n` parameters, you can have full control of your run.
+With the correct use of [`runtime`](#runtime), `runspersub` and `-n` parameters, you can have full control of your run.
- `runtime` defines the _run length_. - `runspersub` defines the maximum number of runs for every [PBS job] submission. @@ -355,52 +350,28 @@ With the correct use of [`runtime`](#runtime), [`runspersub`](#runspersub) and ` Now some practical examples: - **Run 20 years of simulation with resubmission every 5 years**
- To have a _total experiment length_ of 20 years with a 5-year resubmission cycle, leave [`runtime`](#runtime) as the default value of `1 year` and set [`runspersub`](#runspersub) to `5`. Then, run the configuration with `-n` set to `20`: + To have a _total experiment length_ of 20 years with a 5-year resubmission cycle, leave [`runtime`](#runtime) as the default value of `1 year` and set `runspersub` to `5`. Then, run the configuration with `-n` set to `20`: ``` payu run-f -n 20 ``` This will submit subsequent jobs for the following years: 1 to 5, 6 to 10, 11 to 15, and 16 to 20, which is a total of 4 PBS jobs. - **Run 7 years of simulation with resubmission every 3 years**
- To have a _total experiment length_ of 7 years with a 3-year resubmission cycle, leave [`runtime`](#runtime) as the default value of `1 year` and set [`runspersub`](#runspersub) to `3`. Then, run the configuration with `-n` set to `7`: + To have a _total experiment length_ of 7 years with a 3-year resubmission cycle, leave [`runtime`](#runtime) as the default value of `1 year` and set `runspersub` to `3`. Then, run the configuration with `-n` set to `7`: ``` payu run -f -n 7 ``` This will submit subsequent jobs for the following years: 1 to 3, 4 to 6, and 7, which is a total of 3 PBS jobs. -- **Run 3 months and 10 days of simulation in a single submission**
- To have a _total experiment length_ of 3 months and 10 days in a single submission, set the [`runtime`](#runtime) as follows: - ```yaml - years: 0 - months: 3 - days: 10 - ``` - Set [`runspersub`](#runspersub) to `1` (or any value > 1) and run the configuration without `-n` option (or with `-n` set to `1`): - ``` - payu run -f - ``` - -- **Run 1 year and 4 months of simulation with resubmission every 4 months**
- To have a _total experiment length_ of 1 year and 4 months (16 months), you need to split it into multiple runs. For example, 4 runs of 4 months each. In this case, set the [`runtime`](#runtime) as follows: - ```yaml - years: 0 - months: 4 - days: 0 - ``` - Since the _run length_ is set to 4 months, set [`runspersub`](#runspersub) to `1` to resubmit your jobs every 4 months (i.e. every run). Then, run the configuration with `-n` set to `4`: - ``` - payu run -f -n 4 - ``` - - ### Modify PBS resources If the model has been altered and needs more time to complete, more memory, or needs to be submitted under a different NCI project, you will need to modify the following section in the `config.yaml`: -TODO: update with esm config info ```yaml +# PBS configuration + # If submitting to a different project to your default, uncomment line below # and replace PROJECT_CODE with appropriate code. This may require setting shortpath # project: PROJECT_CODE @@ -408,14 +379,14 @@ TODO: update with esm config info # Force payu to always find, and save, files in this scratch project directory # shortpath: /scratch/PROJECT_CODE +# Note: if laboratory is relative path, it is relative to shortpath/$USER +laboratory: access-esm + +jobname: pre-industrial queue: normal -walltime: 3:00:00 -jobname: 1deg_jra55_ryf -mem: 1000GB +walltime: 2:30:00 ``` -TODO: Update with a relevant project - These lines can be edited to change the [PBS directives](https://opus.nci.org.au/display/Help/PBS+Directives+Explained) for the [PBS job][PBS job]. For example, to run {{ model }} under the `ol01` project (COSIMA Working Group), uncomment the line beginning with `# project` by deleting the `#` symbol and replace `PROJECT_CODE` wih `ol01`: @@ -425,13 +396,12 @@ project: ol01 ``` !!! warning - If projects other than `ol01` are used to run {{ model }} configuration, then the `shortpath` field also needs to be uncommented and the path to the desired `/scratch/PROJECT_CODE` added.
+ If a project other than the user's default project is used to run {{ model }} configuration, then the `shortpath` field also needs to be uncommented and the path to the desired `/scratch/PROJECT_CODE` added.
Doing this will make sure the same `/scratch` location is used for the _laboratory_, regardless of which project is used to run the experiment.

To run {{ model }}, you need to be a member of a project with allocated _Service Units (SU)_. For more information, check [how to join relevant NCI projects](/getting_started/first_steps#join-relevant-nci-projects). ### Syncing output data -TODO: Wait for decisions/updates on ESM1.5 syncing. If not part of release, delete this section. The _laboratory_ directory is typically under the `/scratch` storage on _Gadi_, where [files are regularly deleted once they have been unaccessed for a period of time](https://opus.nci.org.au/pages/viewpage.action?pageId=156434436). For this reason climate model outputs need to be moved to a location with longer term storage.
On _Gadi_, this is typically in a folder under a project code on `/g/data`. @@ -439,23 +409,19 @@ On _Gadi_, this is typically in a folder under a project code on `/g/data`. _Payu_ has built-in support to sync outputs, restarts and a copy of the _control_ directory git history to another location.
This feature is controlled by the following section in the `config.yaml` file: ```yaml -# Sync options for automatically copying data from ephemeral scratch space to +# Sync options for automatically copying data from ephemeral scratch space to # longer term storage sync: enable: False # set path below and change to true - path: none # Set to location on /g/data or a remote server and path (rsync syntax) - exclude: - - '*.nc.*' - - 'iceh.????-??-??.nc' + path: null # Set to location on /g/data or a remote server and path (rsync syntax) ``` To enable syncing, change `enable` to `True`, and set `path` to a location on `/g/data`, where _payu_ will copy output and restart folders. A sensible `path` could be: `/g/data/$PROJECT/$USER/{{model}}/experiment_name/`. -!!! admonition tip - The {{model}} default configurations include a [userscript](#userscripts) in the _sync_ step that concatenates daily history/diagnostic output from the Sea-Ice model (CICE5) into monthly files. This speeds up access and saves storage space, but will only run if _sync_ is enabled. +!!! Warning + The {{model}} default configurations include a postprocessing [postcript](#postscripts) which converts atmospheric outputs to NetCDF format and runs in a separate PBS job. This prevents the final run's output and restart files from being automatically synced. Once the postprocessing is completed, the final outputs and restarts can be manually synced by running `payu sync` from the control directory. ### Saving model restarts -TODO: Wait for finalisation of date based restarts for ESM {{ model }} outputs restart files after every run to allow for subsequent runs to start from a previously saved model state.
Restart files can occupy a significant amount of disk space, and keeping a lot of them is often not necessary. @@ -498,39 +464,93 @@ model: access This section specifies the submodels and configuration options required to execute {{ model }} correctly. -Each submodel contains additional configuration options that are read in when the submodel is running. These options are specified in the subfolder of the _control_ directory, whose name matches the submodel's `name` (e.g., configuration options for the `atmosphere` submodel are in the `~/access-esm/esm-pre-industrial/atmosphere` directory). +Each submodel contains additional configuration options that are read in when the submodel is running. These options are specified in the subfolder of the _control_ directory, whose name matches the submodel's `name` (e.g., configuration options for the `atmosphere` submodel are in the `~/access-esm/preindustrial+concentrations/atmosphere` directory). -#TODO: UPDATE WITH UPDATED ESM1.5 CONFIG FILE. ??? code "Expand to show the full `submodels` section" ```yaml submodels: - - name: atmosphere - model: um - ncpus: 192 - exe: /g/data/access/payu/access-esm/bin/coe/um7.3x - input: - - /g/data/access/payu/access-esm/input/pre-industrial/atmosphere - - /g/data/access/payu/access-esm/input/pre-industrial/start_dump
- - name: ocean - model: mom - ncpus: 180 - exe: /g/data/access/payu/access-esm/bin/coe/mom5xx - input: - - /g/data/access/payu/access-esm/input/pre-industrial/ocean/common - - /g/data/access/payu/access-esm/input/pre-industrial/ocean/pre-industrial
- - name: ice - model: cice - ncpus: 12 - exe: /g/data/access/payu/access-esm/bin/coe/cicexx - input: - - /g/data/access/payu/access-esm/input/pre-industrial/ice
- - name: coupler - model: oasis - ncpus: 0 - input: - - /g/data/access/payu/access-esm/input/pre-industrial/coupler + - name: atmosphere + model: um + ncpus: 192 + exe: um_hg3.exe + input: + # Aerosols + - /g/data/vk83/configurations/inputs/access-esm1p5/modern/pre-industrial/atmosphere/aerosol/global.N96/2020.05.19/OCFF_1850_ESM1.anc + - /g/data/vk83/configurations/inputs/access-esm1p5/modern/pre-industrial/atmosphere/aerosol/global.N96/2020.05.19/BC_hi_1850_ESM1.anc + - /g/data/vk83/configurations/inputs/access-esm1p5/modern/pre-industrial/atmosphere/aerosol/global.N96/2020.05.19/scycl_1850_ESM1_v4.anc + - /g/data/vk83/configurations/inputs/access-esm1p5/modern/pre-industrial/atmosphere/aerosol/global.N96/2020.05.19/Bio_1850_ESM1.anc + - /g/data/vk83/configurations/inputs/access-esm1p5/modern/share/atmosphere/aerosol/global.N96/2020.05.19/biogenic_351sm.N96L38 + - /g/data/vk83/configurations/inputs/access-esm1p5/modern/share/atmosphere/aerosol/global.N96/2020.05.19/sulpc_oxidants_N96_L38 + - /g/data/vk83/configurations/inputs/access-esm1p5/modern/share/atmosphere/aerosol/global.N96/2020.05.19/DMS_conc.N96 + # Forcing + - /g/data/vk83/configurations/inputs/access-esm1p5/modern/pre-industrial/atmosphere/forcing/global.N96/2020.05.19/ozone_1850_ESM1.anc + - /g/data/vk83/configurations/inputs/access-esm1p5/modern/pre-industrial/atmosphere/forcing/resolution_independent/2020.05.19/volcts_18502000ave.dat + # Land + - /g/data/vk83/configurations/inputs/access-esm1p5/modern/pre-industrial/atmosphere/land/biogeochemistry/global.N96/2020.05.19/Ndep_1850_ESM1.anc + - /g/data/vk83/configurations/inputs/access-esm1p5/modern/share/atmosphere/land/soiltype/global.N96/2020.05.19/qrparm.soil_igbp_vg + - /g/data/vk83/configurations/inputs/access-esm1p5/modern/share/atmosphere/land/vegetation/global.N96/2020.05.19/cable_vegfunc_N96.anc + - /g/data/vk83/configurations/inputs/access-esm1p5/share/atmosphere/land/biogeochemistry/resolution_independent/2020.05.19/modis_phenology_csiro.txt + - /g/data/vk83/configurations/inputs/access-esm1p5/share/atmosphere/land/biogeochemistry/resolution_independent/2020.05.19/pftlookup_csiro_v16_17tiles_wtlnds.csv + - /g/data/vk83/configurations/inputs/access-esm1p5/share/atmosphere/land/biogeophysics/resolution_independent/2020.05.19/def_soil_params.txt + - /g/data/vk83/configurations/inputs/access-esm1p5/share/atmosphere/land/biogeophysics/resolution_independent/2020.05.19/def_veg_params.txt + # Spectral + - /g/data/vk83/configurations/inputs/access-esm1p5/share/atmosphere/spectral/resolution_independent/2020.05.19/spec3a_sw_hadgem1_6on + - /g/data/vk83/configurations/inputs/access-esm1p5/share/atmosphere/spectral/resolution_independent/2020.05.19/spec3a_lw_hadgem1_6on + # Grids + - /g/data/vk83/configurations/inputs/access-esm1p5/modern/share/atmosphere/grids/global.N96/2020.05.19/qrparm.mask + - /g/data/vk83/configurations/inputs/access-esm1p5/share/atmosphere/grids/resolution_independent/2020.05.19/vertlevs_G3 + # STASH + - /g/data/vk83/configurations/inputs/access-esm1p5/share/atmosphere/stash/2020.05.19/ + + - name: ocean + model: mom + ncpus: 180 + exe: fms_ACCESS-CM.x + input: + # Biogeochemistry + - /g/data/vk83/configurations/inputs/access-esm1p5/modern/share/ocean/biogeochemistry/global.1deg/2020.05.19/dust.nc + - /g/data/vk83/configurations/inputs/access-esm1p5/modern/share/ocean/biogeochemistry/global.1deg/2020.05.19/ocmip2_press_monthly_om1p5_bc.nc + - /g/data/vk83/configurations/inputs/access-esm1p5/share/ocean/biogeochemistry/global.1deg/2024.07.12/bgc_param.nc + - /g/data/vk83/configurations/inputs/access-esm1p5/modern/unused/ocean/biogeochemistry/global.1deg/2020.05.19/ocmip2_fice_monthly_om1p5_bc.nc + - /g/data/vk83/configurations/inputs/access-esm1p5/modern/unused/ocean/biogeochemistry/global.1deg/2020.05.19/ocmip2_xkw_monthly_om1p5_bc.nc + # Tides + - /g/data/vk83/configurations/inputs/access-esm1p5/modern/share/ocean/tides/global.1deg/2020.05.19/roughness_amp.nc + - /g/data/vk83/configurations/inputs/access-esm1p5/modern/share/ocean/tides/global.1deg/2020.05.19/tideamp.nc + # Shortwave + - /g/data/vk83/configurations/inputs/access-esm1p5/modern/share/ocean/shortwave_penetration/global.1deg/2020.05.19/ssw_atten_depth.nc + # Grids + - /g/data/vk83/configurations/inputs/access-esm1p5/modern/share/ocean/grids/mosaic/global.1deg/2020.05.19/grid_spec.nc + + - name: ice + model: cice + ncpus: 12 + exe: cice_access_360x300_12x1_12p.exe + input: + # Grids + - /g/data/vk83/configurations/inputs/access-esm1p5/modern/share/ice/grids/global.1deg/2020.05.19/kmt.nc + - /g/data/vk83/configurations/inputs/access-esm1p5/modern/share/ice/grids/global.1deg/2020.05.19/grid.nc + # Climatology + - /g/data/vk83/configurations/inputs/access-esm1p5/modern/share/ice/climatology/global.1deg/2020.05.19/monthly_sstsss.nc + + - name: coupler + model: oasis + ncpus: 0 + input: + # Grids + - /g/data/vk83/configurations/inputs/access-esm1p5/modern/share/coupler/grids/global.oi_1deg.a_N96/2020.05.19/grids.nc + - /g/data/vk83/configurations/inputs/access-esm1p5/modern/share/coupler/grids/global.oi_1deg.a_N96/2020.05.19/areas.nc + - /g/data/vk83/configurations/inputs/access-esm1p5/modern/share/coupler/grids/global.oi_1deg.a_N96/2020.05.19/masks.nc + # Remapping weights + - /g/data/vk83/configurations/inputs/access-esm1p5/modern/share/coupler/remapping_weights/global.oi_1deg.a_N96/2020.05.19/rmp_cice_to_um1t_CONSERV_FRACNNEI.nc + - /g/data/vk83/configurations/inputs/access-esm1p5/modern/share/coupler/remapping_weights/global.oi_1deg.a_N96/2020.05.19/rmp_um1u_to_cice_CONSERV_FRACNNEI.nc + - /g/data/vk83/configurations/inputs/access-esm1p5/modern/share/coupler/remapping_weights/global.oi_1deg.a_N96/2020.05.19/rmp_um1t_to_cice_CONSERV_DESTAREA.nc + - /g/data/vk83/configurations/inputs/access-esm1p5/modern/share/coupler/remapping_weights/global.oi_1deg.a_N96/2020.05.19/rmp_cice_to_um1u_CONSERV_FRACNNEI.nc + - /g/data/vk83/configurations/inputs/access-esm1p5/modern/share/coupler/remapping_weights/global.oi_1deg.a_N96/2020.05.19/rmp_um1v_to_cice_CONSERV_FRACNNEI.nc + - /g/data/vk83/configurations/inputs/access-esm1p5/modern/share/coupler/remapping_weights/global.oi_1deg.a_N96/2020.05.19/rmp_um1t_to_cice_CONSERV_FRACNNEI.nc + - /g/data/vk83/configurations/inputs/access-esm1p5/modern/share/coupler/remapping_weights/global.oi_1deg.a_N96/2020.05.19/rmp_cice_to_um1v_CONSERV_FRACNNEI.nc + ``` #### Collate @@ -539,15 +559,14 @@ Rather than outputting a single diagnostic file over the whole model horizontal The `collate` section in the `config.yaml` file controls the process that combines these smaller files into a single output file. -TODO: UPDATE WITH CORRECT DATA FOR AN ESM1.5 CONFIG ```yaml +# Collation collate: + exe: mppnccombine.spack restart: true + mem: 4GB walltime: 1:00:00 - mem: 30GB - ncpus: 4 - queue: normal - exe: /g/data/ik11/inputs/access-om2/bin/mppnccombine + mpi: false ``` Restart files are typically tiled in the same way and will also be combined together if the `restart` field is set to `true`. @@ -562,37 +581,27 @@ When running a new configuration, _payu_ automatically commits changes with `git This should not be changed as it is an essential part of the provenance of an experiment.
_payu_ updates the manifest files for every run, and relies on `runlog` to save this information in the `git` history, so there is a record of all inputs, restarts, and executables used in an experiment. -#### Platform - -```yaml -platform: - nodesize: 48 -``` -Set platform-specific default parameters.
-In the example above, the default number of cpus per node is set to 48. -!!! warning - This might need changing if the configuration is run on hardware with different node structure. #### Userscripts -TODO: UPDATE WITH RELEVANT ESM1.5 USERSCRIPTS... WE PROBABLY WON'T BE CALLING ANY IN THE PI RUN... ```yaml userscripts: - error: tools/resub.sh - run: rm -f resubmit.count - sync: /g/data/vk83/apps/om2-scripts/concatenate_ice/concat_ice_daily.sh + # Apply land use changes after each run + run: ./scripts/update_landuse_driver.sh ``` -A dictionary to run scripts or subcommands at various stages of a _payu_ submission. - -- `error` gets called if the model does not run correctly and returns an error code. -- `run` gets called after the model execution, but prior to model output archive. -- `sync` gets called at the start of the sync pbs job. +A dictionary to run scripts or subcommands at various stages of a _payu_ submission. The above example comes from the `release-historical+concentrations` configuration, where the ```update_landuse_driver.sh``` is used to apply historical land use changes at the end of each run. For more information about specific `userscripts` fields, check the relevant section of [_payu_ Configuration Settings documentation](https://payu.readthedocs.io/en/latest/config.html#postprocessing). -### Postscripts -TODO: ADD POSTSCRIPT EXAMPLE ONCE WE MOVE CONVERSION OVER THERE. +#### Postscripts {: id="postscripts"} + +Postprocessing scripts that need to be run after _payu_ has completed all steps that might alter the output directory can be run as postscripts. These run in separate PBS jobs than the main model simulation. + +```yaml +postscript: -v PAYU_CURRENT_OUTPUT_DIR -P ${PROJECT} -lstorage=${PBS_NCI_STORAGE} ./scripts/NetCDF-conversion/UM_conversion_job.sh +``` +All { model } configurations include the above postscript, which converts the atmosphere components output files to NetCDF format, in order to simplify analysis and reduce storage requirements. #### Miscellaneous @@ -608,14 +617,12 @@ qsub_flags: -W umask=027 Each of [{{ model }} components][model components] contains additional configuration options that are read in when the model component is running.
These options are typically useful to modify the physics used in the model, the input data, or the model variables saved in the output files. -TODO: UPDATE WITH CORRECT ESM PATH ONCE NAMING SCHEME CHOSEN - -These configuration options are specified in files located inside a subfolder of the _control_ directory, named according to the submodel's `name` specified in the `config.yaml` `submodels` section (e.g., configuration options for the _ocean_ component are in the `~/access-om2/1deg_jra55_ryf/ocean` directory).
+These configuration options are specified in files located inside a subfolder of the _control_ directory, named according to the submodel's `name` specified in the `config.yaml` `submodels` section (e.g., configuration options for the _ocean_ component are in the `~/access-esm/preindustrial+concentrations/ocean` directory).
To modify these options please refer to the User Guide of the respective model component. ## Get Help -If you have questions or need help regarding {{ model }}, consider creating a topic in the [COSIMA category of the ACCESS Hive Forum](https://forum.access-hive.org.au/c/cosima/29).
+If you have questions or need help regarding {{ model }}, consider creating a topic in the [Earth System Model category of the ACCESS Hive Forum](https://forum.access-hive.org.au/c/esm/earth-system-model/72).
For assistance on how to request help from ACCESS-NRI, follow the [guidelines on how to get help](/about/user_support/#still-need-help). From 822b7d6eab7670ec1d37e7af24bf346a5d58fd1b Mon Sep 17 00:00:00 2001 From: Spencer Wong Date: Wed, 14 Aug 2024 16:30:40 +1000 Subject: [PATCH 05/14] Delete backup of old run tutorial --- .../models/run-a-model/run-access-esm__old.md | 470 ------------------ 1 file changed, 470 deletions(-) delete mode 100644 docs/models/run-a-model/run-access-esm__old.md diff --git a/docs/models/run-a-model/run-access-esm__old.md b/docs/models/run-a-model/run-access-esm__old.md deleted file mode 100644 index 6b4c74e89..000000000 --- a/docs/models/run-a-model/run-access-esm__old.md +++ /dev/null @@ -1,470 +0,0 @@ -{% set model = "ACCESS-ESM" %} -[PBS job]: https://opus.nci.org.au/display/Help/4.+PBS+Jobs - -!!! bug - There is a known issue with the {{ model }} configurations currently set up in the GitHub repository below. - {{ model }} configurations are currently being updated and an ACCESS-NRI release will come - in the following weeks. - If you encounter problems with the current guide in this page, consider asking for help on the [ACCESS-Hive Forum](https://forum.access-hive.org.au). - -# Run {{ model }} - -## Prerequisites - -### General prerequisites - -Before running {{ model }}, you need to fulfil general prerequisites outlined in the [First Steps](/getting_started/first_steps) section. - -If you are unsure whether {{ model }} is the right choice for your experiment, take a look at the overview of [ACCESS Models](/models). - -### Model-specific prerequisites - -{{ model }} is installed on NCI's supercomputer _Gadi_ and uses [_payu_](#payu), a tool for running and managing model experiments. Following these prerequisites ensures you have access to this infrastructure. - -- **Join the _access_ and _hh5_ projects at NCI**
- To join these projects, request membership on the respective [access](https://my.nci.org.au/mancini/project/access/join) and [hh5](https://my.nci.org.au/mancini/project/hh5/join) NCI project pages.
- For more information on joining specific NCI projects, refer to [How to connect to a project](https://opus.nci.org.au/display/Help/How+to+connect+to+a+project). -- **Payu**
- _Payu_ on _Gadi_ is available through the `conda/analysis3` environment in the _hh5_ project.
- After obtaining _hh5_ project membership, load the `conda/analysis3` environment to automatically retrieve _payu_ as follows: - ``` - module use /g/data/hh5/public/modules - module load conda/analysis3 - ``` - To check that _payu_ is available, run: - ``` - payu --version - ``` - - payu --version - 1.0.19 - - -## Get {{ model }} configuration - -{{ model }} configurations are available in the [coecms GitHub](https://github.com/coecms/access-esm) repository.
-To get it on _Gadi_, clone the {{ model }} GitHub repository by running: -``` -git clone https://github.com/coecms/access-esm.git -``` -This will create the `access-esm` folder. - - git clone https://github.com/coecms/access-esm.git - Cloning into 'access-esm'... - remote: Enumerating objects: 1625, done. - remote: Counting objects: 100% (1625/1625), done. - remote: Compressing objects: 100% (575/575), done. - remote: Total 1625 (delta 1042), reused 1621 (delta 1040), pack-reused 0 - Receiving objects: 100% (1625/1625), 2.79 MiB | 11.24 MiB/s, done. - Resolving deltas: 100% (1042/1042), done. - ls ~/access-esm - atmosphere config.yaml coupler ice manifests ocean README.md - - -!!! warning - Some modules may interfere with `git` commands (e.g., matlab/R2018a). If you have trouble cloning the repository, run the following command before trying again: - ``` - module purge - ``` - After this step, don't forget to reload the `conda/analysis3` module to retrieve _payu_, as specified in the [Model-specific prerequisites](#model-specific-prerequisites) section. - -Each {{ model }} configuration is stored as a specially-named branch in the {{ model }} GitHub repository. -To check all the available branches on the repo, run the following command inside the newly-created `access-esm` folder: -``` -git branch -a -``` - - cd ~/access-esm - git branch -a - ٭ main -   remotes/origin/82ka -   remotes/origin/HEAD → origin/main -   remotes/origin/ccarouge-patch-1 -   remotes/origin/historical -   remotes/origin/last-interglacial -   remotes/origin/last-millenium -   remotes/origin/last-millenium-detailed -   remotes/origin/main -   remotes/origin/mid-holocene -   remotes/origin/pre-industrial -   remotes/origin/ssp585 - - -The green-coloured branch (preceded by a star sign `*`) indicates the local branch you are currently in.
-The red-coloured branches are the available remote branches, formatted as `remotes/origin/`.
-To switch to a specific branch you can run the following command: -``` -git checkout -``` -For example, the pre-industrial configuration of {{ model }} is available in the `pre-industrial` branch. To use the pre-industrial configuration, run: -``` -git checkout pre-industrial -``` - - git checkout pre-industrial - branch 'pre-industrial' set up to track 'origin/pre-industrial'. - Switched to a new branch 'pre-industrial' - git branch -   main - ٭ pre-industrial - - -## Edit {{ model }} configuration - -It is good practice to create a new git branch to store all your modifications for a particular run, so as not to modify the reference configuration. - -To create a new branch called _example_run_, as a copy of the `pre-industrial` branch, from within the `access-esm` directory execute: -``` -git checkout -b example_run --no-track origin/pre-industrial -``` - -This command will also switch to the new `example_run` branch. - - git branch -   main - ٭ pre-industrial - git checkout -b example_run --no-track origin/pre-industrial - Switched to a new branch 'example_run' - git branch - ٭ example_run -   main -   pre-industrial - - -### Payu - -[_Payu_](https://payu.readthedocs.io/en/latest) is a workflow management tool for running numerical models in supercomputing environments.
-The general layout of a _payu_-supported model run consists of two main directories: - -- The **laboratory** directory, where all the model components reside. For {{ model }}, it is typically `/scratch/$PROJECT/$USER/access-esm`. -- The **control** directory, where the model configuration resides and from where the model is run (in this example, the cloned directory `~/access-esm`). - -This distinction of directories separates the small-size configuration files from the larger binary outputs and inputs. In this way, the configuration files can be placed in the `$HOME` directory (as it is the only filesystem actively backed-up on _Gadi_), without overloading it with too much data. -Furthermore, this separation allows multiple self-resubmitting experiments that share common executables and input data to be run simultaneously. - -To set up the _laboratory_ directory, run the following command from within the _control_ directory: -``` -payu init -``` -This creates the _laboratory_ directory, together with relevant subdirectories, depending on the configuration. The main subdirectories of interest are: - -- `work` → a temporary directory where the model is run. It gets cleaned after each run. -- `archive` → the directory where output is stored after each run. - - - cd ~/access-esm/esm-pre-industrial - payu init - laboratory path: /scratch/$PROJECT/$USER/access-esm - binary path: /scratch/$PROJECT/$USER/access-esm/bin - input path: /scratch/$PROJECT/$USER/access-esm/input - work path: /scratch/$PROJECT/$USER/access-esm/work - archive path: /scratch/$PROJECT/$USER/access-esm/archive - - -### Edit the _Master Configuration_ file - -The `config.yaml` file located in the _control_ directory, is the _Master Configuration_ file.
-This file, which controls the general model configuration, contains several parts: - -- **PBS resources**{: id="pbs-resources"}
- ```yaml - jobname: pre-industrial - queue: normal - walltime: 3:10:00 - ``` - These lines can be edited to change the [PBS directives](https://opus.nci.org.au/display/Help/PBS+Directives+Explained) for the [PBS job].
- For example, to run {{ model }} under the `tm70` project (ACCESS-NRI), add the following line: - ``` - project: tm70 - ``` - - !!! warning - The `project` entry should always refer to a project with allocated _Service Units (SU)_, that you are a member of. If not set explicitly, {{ model }} will run using your [default project](/getting_started/first_steps#change-default-project-on-gadi) (this default project still needs to have allocated SU). For more information, check [how to join relevant NCI projects](/getting_started/first_steps#join-relevant-nci-projects). - -- **Link to the laboratory directory**
- ```yaml - # note: if laboratory is relative path, it is relative to /scratch/$PROJECT/$USER - laboratory: access-esm - ``` - These lines set the laboratory directory path, which is relative to `/scratch/$PROJECT/$USER`. Absolute paths can also be specified. - -- **Model**
- ```yaml - model: access - ``` - This line tells _payu_ which driver to use for the main model (`access` refers to {{ model }}). - -- **Submodels**
- - ??? code "Expand to show the full `submodels` section" - ```yaml - submodels: - - name: atmosphere - model: um - ncpus: 192 - exe: /g/data/access/payu/access-esm/bin/coe/um7.3x - input: - - /g/data/access/payu/access-esm/input/pre-industrial/atmosphere - - /g/data/access/payu/access-esm/input/pre-industrial/start_dump
- - name: ocean - model: mom - ncpus: 180 - exe: /g/data/access/payu/access-esm/bin/coe/mom5xx - input: - - /g/data/access/payu/access-esm/input/pre-industrial/ocean/common - - /g/data/access/payu/access-esm/input/pre-industrial/ocean/pre-industrial
- - name: ice - model: cice - ncpus: 12 - exe: /g/data/access/payu/access-esm/bin/coe/cicexx - input: - - /g/data/access/payu/access-esm/input/pre-industrial/ice
- - name: coupler - model: oasis - ncpus: 0 - input: - - /g/data/access/payu/access-esm/input/pre-industrial/coupler - ``` - - {{ model }} is a coupled model deploying multiple submodels (i.e. [model components](/models/configurations/access-esm/#model-components)). - This section specifies the submodels and configuration options required to execute the model correctly.
- Each submodel contains additional configuration options that are read in when the submodel is running. These options are specified in the subfolder of the _control_ directory, whose name matches the submodel's `name` (e.g., configuration options for the `atmosphere` submodel are in the `~/access-esm/esm-pre-industrial/atmosphere` directory). - -- **Collate**
- ```yaml - collate: - exe: /g/data/access/payu/access-esm/bin/mppnccombine - restart: true - mem: 4GB - ``` - The `collate` process combines a number of smaller files, which contain different parts of the model grid, into target output files. Restart files are typically tiled in the same way and will also be combined together if the `restart` option is set to `true`. - -- **Restart**
- ```yaml - restart: /g/data/access/payu/access-esm/restart/pre-industrial - ``` - This is the location of the files used for a warm restart. - -- **Start date and run length**{: id="runtime"}
- ```yaml - calendar: - start: - year: 101 - month: 1 - days: 1 - runtime: - years: 1 - months: 0 - days: 0 - ``` - This section specifies the start date and run length. - - !!! warning - The _run length_ (controlled by [`runtime`](#runtime)) can be different from the _total experiment length_. Also, while `runtime` can be reduced, it should not be increased to more than 1 year to avoid errors. For more information about the difference between _run length_ and _total experiment length_, or how to run the model for more than 1 year, refer to the section [Run configuration for multiple years](#run-configuration-for-multiple-years). - -- **Number of runs per PBS submission**{: id="runspersub"}
- ```yaml - runspersub: 1 - ``` - {{ model }} configurations are often run in multiple steps (or cycles), with _payu_ running a maximum of `runspersub` runs for every [PBS job] submission. - - !!! warning - If you increase `runspersub`, you may need to increase the `walltime` in the [PBS resources](#pbs-resources). - -To find out more about other configuration settings for the `config.yaml` file, check out [how to configure your experiment with _payu_](https://payu.readthedocs.io/en/latest/config.html). - -### Edit a single {{ model }} component configuration - -Each of [{{ model }} components](/models/configurations/access-esm/#model-components) contains additional configuration options that are read in when the model component is running.
-These options are typically useful to modify the physics used in the model, the input data, or the model variables saved in the output files. - -These configuration options are specified in files located inside a subfolder of the _control_ directory, named according to the submodel's `name` specified in the `config.yaml` `submodels` section (e.g., configuration options for the `atmosphere` submodel are in the `~/access-esm/esm-pre-industrial/atmosphere` directory).
-To modify these options please refer to the User Guide of the respective model component. - -## Run {{ model }} configuration - -After editing the configuration, you are ready to run {{ model }}.
-{{ model }} suites run on [_Gadi_](https://opus.nci.org.au/display/Help/0.+Welcome+to+Gadi#id-0.WelcometoGadi-Overview) through a [PBS job] submission managed by _payu_. - -### Payu setup (optional) - -As a first step, from within the _control_ directory, it is good practice to run: -``` -payu setup -``` - -This will prepare the model run, based on the experiment configuration. - - payu setup - laboratory path: /scratch/$PROJECT/$USER/access-esm - binary path: /scratch/$PROJECT/$USER/access-esm/bin - input path: /scratch/$PROJECT/$USER/access-esm/input - work path: /scratch/$PROJECT/$USER/access-esm/work - archive path: /scratch/$PROJECT/$USER/access-esm/archive - Loading input manifest: manifests/input.yaml - Loading restart manifest: manifests/restart.yaml - Loading exe manifest: manifests/exe.yaml - Setting up atmosphere - Setting up ocean - Setting up ice - Setting up coupler - Checking exe and input manifests - Updating full hashes for 3 files in manifests/exe.yaml - Creating restart manifest - Updating full hashes for 30 files in manifests/restart.yaml - Writing manifests/restart.yaml - Writing manifests/exe.yaml - - -!!! tip - This step can be skipped as it is also included in the run command. However, running it explicitly helps to check for errors and make sure executable and restart directories are accessible. - -### Run configuration - -To run {{ model }} configuration for one _run length_ (controlled by [`runtime`](#runtime) in the `config.yaml` file), from within the _control_ directory execute: - -``` -payu run -f -``` - -This will submit a single job to the queue with a _total experiment length_ of `runtime`. If there is no previous run, it will start from the [`start`](#runtime) date indicated in the `config.yaml` file. Otherwise, it will perform a warm restart from a previously saved restart file.
- -!!! tip - The `-f` option ensures that _payu_ will run even if there is an existing non-empty `work` directory created from a previous failed run. - - - payu run -f - Loading input manifest: manifests/input.yaml - Loading restart manifest: manifests/restart.yaml - Loading exe manifest: manifests/exe.yaml - payu: Found modules in /opt/Modules/v4.3.0 - qsub -q normal -P <project> -l walltime=11400 -l ncpus=384 -l mem=1536GB -N pre-industrial -l wd -j n -v PAYU_PATH=/g/data/hh5/public/apps/miniconda3/envs/analysis3-23.01/bin,MODULESHOME=/opt/Modules/v4.3.0,MODULES_CMD=/opt/Modules/v4.3.0/libexec/modulecmd.tcl,MODULEPATH=/g/data3/hh5/public/modules:/etc/scl/modulefiles:/opt/Modules/modulefiles:/opt/Modules/v4.3.0/modulefiles:/apps/Modules/modulefiles -W umask=027 -l storage=gdata/access+gdata/hh5 -- /g/data/hh5/public/apps/miniconda3/envs/analysis3-23.01/bin/python3.9 /g/data/hh5/public/apps/miniconda3/envs/analysis3-23.01/bin/payu-run - <job-ID>.gadi-pbs - - -### Run configuration for multiple years - -If you want to run {{ model }} configuration for multiple _run lengths_ (controlled by [`runtime`](#runtime) in the `config.yaml` file), use the option `-n`: - -``` -payu run -f -n -``` - -This will run the configuration `number-of-runs` times with a _total experiment length_ of `runtime * number-of-runs`. The number of consecutive [PBS jobs][PBS job] submitted to the queue depends on the [`runspersub`](#runspersub) value specified in the config.yaml file. - -### Understand `runtime`, `runspersub`, and `-n` parameters - -With the correct use of [`runtime`](#runtime), [`runspersub`](#runspersub) and `-n` parameters, you can have full control of your run.
- -- `runtime` defines the _run length_. -- `runspersub` defines the maximum number of runs for every [PBS job] submission. -- `-n` sets the number of runs to be performed. - -Now some practical examples: - -- **Run 20 years of simulation with resubmission every 5 years**
- To have a _total experiment length_ of 20 years with a 5-year resubmission cycle, leave [`runtime`](#runtime) as the default value of `1 year` and set [`runspersub`](#runspersub) to `5`. Then, run the configuration with `-n` set to `20`: - ``` - payu run-f -n 20 - ``` - This will submit subsequent jobs for the following years: 1 to 5, 6 to 10, 11 to 15, and 16 to 20, which is a total of 4 PBS jobs. - -- **Run 7 years of simulation with resubmission every 3 years**
- To have a _total experiment length_ of 7 years with a 3-year resubmission cycle, leave [`runtime`](#runtime) as the default value of `1 year` and set [`runspersub`](#runspersub) to `3`. Then, run the configuration with `-n` set to `7`: - ``` - payu run -f -n 7 - ``` - This will submit subsequent jobs for the following years: 1 to 3, 4 to 6, and 7, which is a total of 3 PBS jobs. - -- **Run 3 months and 10 days of simulation in a single submission**
- To have a _total experiment length_ of 3 months and 10 days in a single submission, set the [`runtime`](#runtime) as follows: - ```yaml - years: 0 - months: 3 - days: 10 - ``` - Set [`runspersub`](#runspersub) to `1` (or any value > 1) and run the configuration without `-n` option (or with `-n` set to `1`): - ``` - payu run -f - ``` - -- **Run 1 year and 4 months of simulation with resubmission every 4 months**
- To have a _total experiment length_ of 1 year and 4 months (16 months), you need to split it into multiple runs. For example, 4 runs of 4 months each. In this case, set the [`runtime`](#runtime) as follows: - ```yaml - years: 0 - months: 4 - days: 0 - ``` - Since the _run length_ is set to 4 months, set [`runspersub`](#runspersub) to `1` to resubmit your jobs every 4 months (i.e. every run). Then, run the configuration with `-n` set to `4`: - ``` - payu run -f -n 4 - ``` - -## Monitor {{ model }} runs - -You can execute the following command to show the status of all your submitted [PBS jobs][PBS job]: - -``` -qstat -u $USER -``` - - qstat -u $USER - Job id                Name             User             Time Use S Queue - --------------------- ---------------- ---------------- -------- - ----- - <job-ID>.gadi-pbs     pre-industrial   <$USER>            <time> R normal-exec - <job-ID-2>.gadi-pbs   <other-job-name> <$USER>            <time> R normal-exec - <job-ID-3>.gadi-pbs   <other-job-name> <$USER>            <time> R normal-exec - - -If you changed the `jobname` in the [PBS resources](#pbs-resources) of the _Master Configuration_ file, that will appear as your job's _Name_ instead of `pre-industrial`.
- -_S_ indicates the status of your run, where: - -- _Q_ → Job waiting in the queue to start -- _R_ → Job running -- _E_ → Job ending -- _H_ → Job on hold - -If there are no jobs listed with your `jobname` (or if no job is listed), your run either successfully completed or was terminated due to an error.
-For more information, check [NCI documentation](https://opus.nci.org.au/display/Help/FAQ+1%3A+Why+My+Jobs+are+NOT+Running). - -### Stop a run - -If you want to manually terminate a run, you can do so by executing: -``` -qdel -``` - -### Error and output log files - -While the model is running, _payu_ saves the standard output and standard error in the respective `access.out` and `access.err` files in the _control_ directory. You can examine the contents of these files to check on the status of a run as it progresses.
-When the model completes its run, or if it crashes, the output and error log files are by default renamed as `jobname.o` and `jobname.e`, respectively. - -### Model Live Diagnostics - -ACCESS-NRI developed the [Model Live Diagnostics](/model_evaluation/model_diagnostics) framework to check, monitor, visualise, and evaluate model behaviour and progress of ACCESS models currently running on _Gadi_.
-For a complete documentation on how to use this framework, check the [Model Diagnostics documentation](https://med-live-diagnostics.readthedocs.io/en/latest/index.html). - -## {{ model }} outputs - -At the end of the model run, output files (and restart files) are moved from the `work` directory into the `archive` directory under `/scratch/$PROJECT/$USER/access-esm/archive/access-esm`, where they are further subdivided for each run. They are also symlinked in the _control_ directory to `~/access-esm/archive`.
-The naming format for a typical output folder is `outputXXX` and for a restart folder `restartXXX`, where _XXX_ is the run number starting from `000`. - -!!! tip - A run with a different {{ model }} configuration (different git branch) counts as a new run.
- Thus, if output folders already exist, the internal number of the new output folder will be set to the first available _XXX_ number. - -Outputs and restarts are separated in the respective folders for each model component.
-For the atmospheric output data, the files are usually [UM fieldsfile](https://code.metoffice.gov.uk/doc/um/latest/papers/umdp_F03.pdf), formatted as `a.p`. - - cd /scratch/$PROJECT/$USER/access-esm/archive/esm-pre-industrial - ls - output000 pbs_logs restart000 - ls output000/atmosphere - aiihca.daa1210 aiihca.daa1810 aiihca.paa1apr aiihca.paa1jun aiihca.pea1apr aiihca.pea1jun aiihca.pga1apr aiihca.pga1jun atm.fort6.pe0 exstat ihist prefix.CNTLGEN UAFLDS_A aiihca.daa1310 aiihca.daa1910 aiihca.paa1aug aiihca.paa1mar aiihca.pea1aug aiihca.pea1mar aiihca.pga1aug aiihca.pga1mar cable.nml fort.57 INITHIS prefix.PRESM_A um_env.py aiihca.daa1410 aiihca.daa1a10 aiihca.paa1dec aiihca.paa1may aiihca.pea1dec aiihca.pea1may aiihca.pga1dec aiihca.pga1may CNTLALL ftxx input_atm.nml SIZES xhist aiihca.daa1510 aiihca.daa1b10 aiihca.paa1feb aiihca.paa1nov aiihca.pea1feb aiihca.pea1nov aiihca.pga1feb aiihca.pga1nov CONTCNTL ftxx.new namelists STASHC aiihca.daa1610 aiihca.daa1c10 aiihca.paa1jan aiihca.paa1oct aiihca.pea1jan aiihca.pea1oct aiihca.pga1jan aiihca.pga1oct debug.root.01 ftxx.vars nout.000000 thist aiihca.daa1710 aiihca.daa2110 aiihca.paa1jul aiihca.paa1sep aiihca.pea1jul aiihca.pea1sep aiihca.pga1jul aiihca.pga1sep errflag hnlist prefix.CNTLATM UAFILES_A - - - -- [https://github.com/coecms/esm-pre-industrial](https://github.com/coecms/esm-pre-industrial) -- [https://payu.readthedocs.io/en/latest/usage.html](https://payu.readthedocs.io/en/latest/usage.html) - From 88b6511b4048e205c7e8db576a6fc2060d7ada3e Mon Sep 17 00:00:00 2001 From: Spencer Wong Date: Thu, 22 Aug 2024 09:53:16 +1000 Subject: [PATCH 06/14] New ESM information --- docs/models/run-a-model/run-access-esm.md | 75 ++++++++++++++++------- 1 file changed, 52 insertions(+), 23 deletions(-) diff --git a/docs/models/run-a-model/run-access-esm.md b/docs/models/run-a-model/run-access-esm.md index 0ba18deb4..b16b0dcfd 100644 --- a/docs/models/run-a-model/run-access-esm.md +++ b/docs/models/run-a-model/run-access-esm.md @@ -205,7 +205,7 @@ qdel which kills the specified job without waiting for it to complete. !!! tip - If you started an {{ model }} run using the `-n` option (e.g., to [run the model for more than 5 years](#run-configuration-for-more-than-5-years)), but subsequently decide not to keep running after the current process completes, you can create a file called `stop_run` in the _control_ directory.
+ If you started an {{ model }} run using the `-n` option (e.g., to [run the model for several years](#multiple-runs)), but subsequently decide not to keep running after the current process completes, you can create a file called `stop_run` in the _control_ directory.
This will prevent _payu_ from submitting another job. ### Error and output log files @@ -303,10 +303,9 @@ The `config.yaml` file located in the _control_ directory is the _Master Configu To find out more about configuration settings for the `config.yaml` file, refer to [how to configure your experiment with payu](https://payu.readthedocs.io/en/latest/config.html). -### Change run length {: id="runtime"} +### Run lengths {: id="runtime"} -One of the most common changes is to adjust the duration of the model run.
-For example, when debugging changes to a model, it is common to reduce the run length to minimise resource consumption and return faster feedback on changes. +One of the most common changes is to adjust the duration of the model run.
{{model}} simulations are split into smaller _run lengths_, each with the duration specified by the `runtime` settings in the `config.yaml` file: The length of an {{model}} run is controlled by the `runtime` settings in the `config.yaml` file: @@ -316,32 +315,22 @@ The length of an {{model}} run is controlled by the `runtime` settings in the `c months: 0 days: 0 ``` - -For example, to make the model run for a single month, change the `runtime` to -``` - runtime: - years: 0 - months: 1 - days: 0 -``` -!!! warning - The sea ice component of {{ model }} is configured to produce restart files only at the end of each year. While simulations shorter than a year can be helpful for troubleshooting, they will not produce valid restart files and it will not be possible to continue them after the first run. If running more than one segment is required, the `runtime` setting for {{ model }} should be left at one year. +At the end of each run length, each model component saves its state into a _restart file_, allowing the simulation to be continued in subsequent runs. !!! warning - The _run length_ (controlled by `runtime`) can be different from the _total experiment length_. While `runtime` can be reduced, it should not be increased to more than 1 year to avoid errors. For more information about the difference between _run length_ and _total experiment length_, or how to run the model for more than 1 year, refer to the section [Understand `runtime`, `runspersub`, and `-n` parameters](#multiple-runs). - -If you want to run {{ model }} configuration for multiple _run lengths_ (each with duration [`runtime`](#runtime) in the `config.yaml` file), use the option `-n`: + The _run length_ (controlled by `runtime`) should be left at 1 year for {{model}} experiments in order to avoid errors. Simulations longer than 1 year can be set up by running for multiple run lengths, as described in the section [Understand `runtime`, `runspersub`, and `-n` parameters](#multiple-runs). Shorter simulations can also be useful when setting up and debugging new experiments, however require additional model settings to be changed. See the section [Running for less than one year](#shorter-runs) for details. +To run {{ model }} configuration for multiple subsequent _run lengths_ (each with duration `runtime` in the `config.yaml` file), use the option `-n` with the `payu run` command: ``` payu run -f -n ``` -This will run the configuration `number-of-runs` times with a _total experiment length_ of `runtime * number-of-runs`. The number of consecutive [PBS jobs][PBS job] submitted to the queue depends on the [`runspersub`](#runspersub) value specified in the config.yaml file. - +This will run the configuration `number-of-runs` times, resulting in a _total experiment length_ of `runtime * number-of-runs`. The runs will be split accross a number of consecutive [PBS jobs][PBS job] submitted to the queue, as controlled by the `runspersub` value specified in the config.yaml file. + ### Understand `runtime`, `runspersub`, and `-n` parameters {: id="multiple-runs"} -With the correct use of [`runtime`](#runtime), `runspersub` and `-n` parameters, you can have full control of your run.
+With the correct use of [`runtime`](#runtime), `runspersub` and `-n` parameters, you can have full control of your experiment.
- `runtime` defines the _run length_. - `runspersub` defines the maximum number of runs for every [PBS job] submission. @@ -363,6 +352,17 @@ Now some practical examples: ``` This will submit subsequent jobs for the following years: 1 to 3, 4 to 6, and 7, which is a total of 3 PBS jobs. +### Running for less than one year {: id="shorter-runs"} +When debugging changes to a model, it is common to reduce the run length to minimise resource consumption and return faster feedback on changes. In order to run the model for a single month, the `runtime` can be changed to + +``` + runtime: + years: 0 + months: 1 + days: 0 +``` + +With the default configuration settings, the sea ice component of {{ model }} will produce restart files only at the end of each year. If valid restart files are required when running shorter simulations, the sea ice model configuration should be modified so that restart files are produced at monthly frequencies. To do this, change the `dumpfreq = 'y'` setting to `dumpfreq = 'm'` in the `cice_in.nml` configuration file located in the `ice` control directory. ### Modify PBS resources @@ -601,7 +601,19 @@ Postprocessing scripts that need to be run after _payu_ has completed all steps ```yaml postscript: -v PAYU_CURRENT_OUTPUT_DIR -P ${PROJECT} -lstorage=${PBS_NCI_STORAGE} ./scripts/NetCDF-conversion/UM_conversion_job.sh ``` -All { model } configurations include the above postscript, which converts the atmosphere components output files to NetCDF format, in order to simplify analysis and reduce storage requirements. +All {{ model }} configurations include an automatic NetCDF conversion postscript, which converts the atmosphere model's fields file format output to NetCDF in order to aid analysis and reduce storage requirements. By default, both the original atmospheric fields files, and the NetCDF files will be retained at the end of a simulation. The conversion script has an automatic deletion option, which deletes the original fields files upon successful conversion. This setting is deactivated by default, however can be enabled in the following line of the job submission script `./scripts/NetCDF-conversion/UM_conversion_job.sh`: + +```bash +esm1p5_convert_nc $PAYU_CURRENT_OUTPUT_DIR +``` + +To activate automatic fields file deletion, add a `--delete-ff` flag to the end of the line: + +```bash +esm1p5_convert_nc $PAYU_CURRENT_OUTPUT_DIR --delete-ff +``` + +It's recommended to run an experiment for a few years and checking that the conversion is working as desired before activating the automatic deletion. #### Miscellaneous @@ -614,12 +626,29 @@ qsub_flags: -W umask=027 ### Edit a single {{ model }} component configuration -Each of [{{ model }} components][model components] contains additional configuration options that are read in when the model component is running.
-These options are typically useful to modify the physics used in the model, the input data, or the model variables saved in the output files. +Each of [{{ model }} components][model components] contains additional configuration options that are read in when the model component is running.
These options are typically useful to modify the physics used in the model, the input data, or the model variables saved in the output files. These configuration options are specified in files located inside a subfolder of the _control_ directory, named according to the submodel's `name` specified in the `config.yaml` `submodels` section (e.g., configuration options for the _ocean_ component are in the `~/access-esm/preindustrial+concentrations/ocean` directory).
To modify these options please refer to the User Guide of the respective model component. +### Controlling model output +Selecting the variables to save from a simulation can be a balance between enabling future analyisis and minimising storage requirements. The choice and frequency of variables saved by each model can be configured from within each submodel's control directory. + +Each submodel's control directory contains _detailed_ and _standard_ presets for controling the output, located in the `diagnostic_profiles` subdirectories (e.g. `~/access-esm/preindustrial+concentrations/ice/diagnostic_profiles` for the sea ice submodel). The _detailed_ profiles request a large number of variables at higher frequencies, while the _standard_ profiles restrict the output to variables more regularly used accross the community. + +Selecting a preset output profile to use in a simulation can be done by pointing the following symbolic links to the desired profile: + + * `STASHC` in the atmosphere control directory. + * `diag_table` in the ocean control directory. + * `ice_history.nml` in the ice control directory. + +For example, to select the _detailed_ output profile for the atmosphere: + + cd ~/access-esm/preindustrial+concentrations/atmosphere + ln -sf diagnostic_profiles/STASHC_detailed STASHC + + + ## Get Help If you have questions or need help regarding {{ model }}, consider creating a topic in the [Earth System Model category of the ACCESS Hive Forum](https://forum.access-hive.org.au/c/esm/earth-system-model/72).
From f47f02ba0cae810d6dd39ac8ac68e42baf5ad213 Mon Sep 17 00:00:00 2001 From: Spencer Wong Date: Thu, 22 Aug 2024 10:00:26 +1000 Subject: [PATCH 07/14] Add link to release post --- docs/models/run-a-model/run-access-esm.md | 4 +--- 1 file changed, 1 insertion(+), 3 deletions(-) diff --git a/docs/models/run-a-model/run-access-esm.md b/docs/models/run-a-model/run-access-esm.md index b16b0dcfd..e2179bca8 100644 --- a/docs/models/run-a-model/run-access-esm.md +++ b/docs/models/run-a-model/run-access-esm.md @@ -20,8 +20,7 @@ If you are unsure whether {{ model }} is the right choice for your experiment, t All {{model}} configurations are open source, licensed under [CC BY 4.0](https://creativecommons.org/licenses/by/4.0/?ref=chooser-v1")![CC icon](https://mirrors.creativecommons.org/presskit/icons/cc.svg?ref=chooser-v1){: style="height:1em;margin-left:0.2em;vertical-align:text-top;"}![BY icon](https://mirrors.creativecommons.org/presskit/icons/by.svg?ref=chooser-v1){: style="height:1em;margin-left:0.2em;vertical-align:text-top;"} and available on [ACCESS-NRI GitHub]({{github_configs}}). - -{{ model }} release notes are [available on the ACCESS-Hive Forum ](MISSING LINK) and are updated when new releases are made available. +{{ model }} release notes are [available on the ACCESS-Hive Forum ](https://forum.access-hive.org.au/t/access-esm1-5-release-information/2352) and are updated when new releases are made available. ## Prerequisites @@ -595,7 +594,6 @@ For more information about specific `userscripts` fields, check the relevant sec #### Postscripts {: id="postscripts"} - Postprocessing scripts that need to be run after _payu_ has completed all steps that might alter the output directory can be run as postscripts. These run in separate PBS jobs than the main model simulation. ```yaml From e450e2db2c068f9eff9e22669be12645ca828a1f Mon Sep 17 00:00:00 2001 From: Spencer Wong Date: Thu, 22 Aug 2024 10:37:04 +1000 Subject: [PATCH 08/14] Spelling errors --- docs/models/run-a-model/run-access-esm.md | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/docs/models/run-a-model/run-access-esm.md b/docs/models/run-a-model/run-access-esm.md index e2179bca8..31bb4660b 100644 --- a/docs/models/run-a-model/run-access-esm.md +++ b/docs/models/run-a-model/run-access-esm.md @@ -213,7 +213,7 @@ which kills the specified job without waiting for it to complete. When the model completes a run, PBS writes the standard output and error streams to two files inside the _control_ directory: `.o` and `.e`, respectively. These files usually contain logs about _payu_ tasks, and give an overview of the resources used by the job.
-To move these files to the `archive` directory, use the following commmand: +To move these files to the `archive` directory, use the following command: ``` payu sweep ``` @@ -325,7 +325,7 @@ To run {{ model }} configuration for multiple subsequent _run lengths_ (each wit payu run -f -n ``` -This will run the configuration `number-of-runs` times, resulting in a _total experiment length_ of `runtime * number-of-runs`. The runs will be split accross a number of consecutive [PBS jobs][PBS job] submitted to the queue, as controlled by the `runspersub` value specified in the config.yaml file. +This will run the configuration `number-of-runs` times, resulting in a _total experiment length_ of `runtime * number-of-runs`. The runs will be split across a number of consecutive [PBS jobs][PBS job] submitted to the queue, as controlled by the `runspersub` value specified in the config.yaml file. ### Understand `runtime`, `runspersub`, and `-n` parameters {: id="multiple-runs"} @@ -388,7 +388,7 @@ walltime: 2:30:00 These lines can be edited to change the [PBS directives](https://opus.nci.org.au/display/Help/PBS+Directives+Explained) for the [PBS job][PBS job]. -For example, to run {{ model }} under the `ol01` project (COSIMA Working Group), uncomment the line beginning with `# project` by deleting the `#` symbol and replace `PROJECT_CODE` wih `ol01`: +For example, to run {{ model }} under the `ol01` project (COSIMA Working Group), uncomment the line beginning with `# project` by deleting the `#` symbol and replace `PROJECT_CODE` with `ol01`: ```yaml project: ol01 @@ -630,9 +630,9 @@ These configuration options are specified in files located inside a subfolder of To modify these options please refer to the User Guide of the respective model component. ### Controlling model output -Selecting the variables to save from a simulation can be a balance between enabling future analyisis and minimising storage requirements. The choice and frequency of variables saved by each model can be configured from within each submodel's control directory. +Selecting the variables to save from a simulation can be a balance between enabling future analysis and minimising storage requirements. The choice and frequency of variables saved by each model can be configured from within each submodel's control directory. -Each submodel's control directory contains _detailed_ and _standard_ presets for controling the output, located in the `diagnostic_profiles` subdirectories (e.g. `~/access-esm/preindustrial+concentrations/ice/diagnostic_profiles` for the sea ice submodel). The _detailed_ profiles request a large number of variables at higher frequencies, while the _standard_ profiles restrict the output to variables more regularly used accross the community. +Each submodel's control directory contains _detailed_ and _standard_ presets for controlling the output, located in the `diagnostic_profiles` subdirectories (e.g. `~/access-esm/preindustrial+concentrations/ice/diagnostic_profiles` for the sea ice submodel). The _detailed_ profiles request a large number of variables at higher frequencies, while the _standard_ profiles restrict the output to variables more regularly used across the community. Selecting a preset output profile to use in a simulation can be done by pointing the following symbolic links to the desired profile: From 625aa206f546d51b767e2769bb63339dd73a44cb Mon Sep 17 00:00:00 2001 From: Spencer Wong <88933912+blimlim@users.noreply.github.com> Date: Thu, 22 Aug 2024 13:29:38 +1000 Subject: [PATCH 09/14] Apply suggestions from code review Review suggestions Co-authored-by: Aidan Heerdegen --- docs/models/configurations/access-esm.md | 6 ++--- docs/models/run-a-model/run-access-esm.md | 30 ++++++++++++----------- 2 files changed, 19 insertions(+), 17 deletions(-) diff --git a/docs/models/configurations/access-esm.md b/docs/models/configurations/access-esm.md index 8034e5ee4..2c3dad2dd 100644 --- a/docs/models/configurations/access-esm.md +++ b/docs/models/configurations/access-esm.md @@ -14,11 +14,11 @@ ACCESS-NRI has released ACCESS-ESM1.5 configurations as an adaptation of those o There are currently two supported configurations: - ***Pre-industrial concentration driven***:
- A global coupled model configuration running in co2 concentration driven mode under pre-industrial forcings, as described in [Ziehn et al. (2020)](https://doi.org/10.1071/ES19035). - Pre-industrial forcing data including atmospheric co2 concentrations are primarily sourced from UKMO versions of CMIP6 inputs, with additional atmospheric forcings sourced from CMIP5 and land cover data adapted from [Lawrence et al. (2012)](https://doi.org/10.1175/JCLI-D-11-00256.1). + A global coupled model configuration running in CO~2~ concentration driven mode under pre-industrial forcings, as described in [Ziehn et al. (2020)](https://doi.org/10.1071/ES19035). + Pre-industrial forcing data including atmospheric CO~2~ concentrations are primarily sourced from UKMO versions of CMIP6 inputs, with additional atmospheric forcings sourced from CMIP5 and land cover data adapted from [Lawrence et al. (2012)](https://doi.org/10.1175/JCLI-D-11-00256.1). - ***Historical concentration driven***:
A global coupled model configuration running in co2 concentration driven mode under time varying historical (1850-2014) forcings, as described in [Ziehn et al. (2020)](https://doi.org/10.1071/ES19035). - Historical forcing data including atmospheric co2 concentrations are primarily sourced from UKMO versions of CMIP6 inputs, with land use change data adapted from the Land-Use Harmonisation 2 (LUH2) dataset developed for CMIP6 [(Hurtt et al. 2017)](https://doi.org/10.22033/ESGF/input4MIPs.1127). + Historical forcing data including atmospheric CO~2~ concentrations are primarily sourced from UKMO versions of CMIP6 inputs, with land use change data adapted from the Land-Use Harmonisation 2 (LUH2) dataset developed for CMIP6 [(Hurtt et al. 2017)](https://doi.org/10.22033/ESGF/input4MIPs.1127). ### Model components - **Atmoshpere**: [UM7.3](/models/model_components/atmosphere#unified-model-um), GA7.1 science configuration.
diff --git a/docs/models/run-a-model/run-access-esm.md b/docs/models/run-a-model/run-access-esm.md index 31bb4660b..9f8652db3 100644 --- a/docs/models/run-a-model/run-access-esm.md +++ b/docs/models/run-a-model/run-access-esm.md @@ -75,7 +75,7 @@ To clone this branch to a location on _Gadi_, run: cd ~/access-esm1.5 payu clone -b expt -B release-preindustrial+concentrations {{ github_configs }} preindustrial+concentrations -In the example above the `payu clone` command clones the concentration driven pre-industrial configuration (` -B -B release-preindustrial+concentrations`) +In the example above the `payu clone` command clones the concentration driven pre-industrial configuration (`-B release-preindustrial+concentrations`) to a new experiment branch (`-b expt`) to a directory named `preindustrial+concentrations`. !!! admonition tip Anyone using a configuration is advised to clone only a single branch (as shown in the example above) and not the entire repository. @@ -317,7 +317,7 @@ The length of an {{model}} run is controlled by the `runtime` settings in the `c At the end of each run length, each model component saves its state into a _restart file_, allowing the simulation to be continued in subsequent runs. !!! warning - The _run length_ (controlled by `runtime`) should be left at 1 year for {{model}} experiments in order to avoid errors. Simulations longer than 1 year can be set up by running for multiple run lengths, as described in the section [Understand `runtime`, `runspersub`, and `-n` parameters](#multiple-runs). Shorter simulations can also be useful when setting up and debugging new experiments, however require additional model settings to be changed. See the section [Running for less than one year](#shorter-runs) for details. + The _run length_ (controlled by `runtime`) should be left at 1 year for {{model}} experiments in order to avoid errors. Shorter simulations can be useful when setting up and debugging new experiments, however they require additional configuration changes. See the section [Running for less than one year](#shorter-runs) for details. To run {{ model }} configuration for multiple subsequent _run lengths_ (each with duration `runtime` in the `config.yaml` file), use the option `-n` with the `payu run` command: @@ -329,28 +329,30 @@ This will run the configuration `number-of-runs` times, resulting in a _total ex ### Understand `runtime`, `runspersub`, and `-n` parameters {: id="multiple-runs"} -With the correct use of [`runtime`](#runtime), `runspersub` and `-n` parameters, you can have full control of your experiment.
+It is possible to have more than one model run per queue submit. With the correct use of [`runtime`](#runtime), `runspersub` and `-n` parameters, you can have full control of your experiment.
- `runtime` defines the _run length_. - `runspersub` defines the maximum number of runs for every [PBS job] submission. +- `walltime` defines the maximum time of every [PBS job] submission. - `-n` sets the number of runs to be performed. Now some practical examples: - **Run 20 years of simulation with resubmission every 5 years**
- To have a _total experiment length_ of 20 years with a 5-year resubmission cycle, leave [`runtime`](#runtime) as the default value of `1 year` and set `runspersub` to `5`. Then, run the configuration with `-n` set to `20`: + To have a _total experiment length_ of 20 years with a 5-year resubmission cycle, leave [`runtime`](#runtime) as the default value of `1 year`, set `runspersub` to `5` and `walltime` to `10:00:00`. Then, run the configuration with `-n` set to `20`: ``` payu run-f -n 20 ``` This will submit subsequent jobs for the following years: 1 to 5, 6 to 10, 11 to 15, and 16 to 20, which is a total of 4 PBS jobs. - **Run 7 years of simulation with resubmission every 3 years**
- To have a _total experiment length_ of 7 years with a 3-year resubmission cycle, leave [`runtime`](#runtime) as the default value of `1 year` and set `runspersub` to `3`. Then, run the configuration with `-n` set to `7`: + To have a _total experiment length_ of 7 years with a 3-year resubmission cycle, leave [`runtime`](#runtime) as the default value of `1 year`, set `runspersub` to `3` and `walltime` to `6:00:00`. Then, run the configuration with `-n` set to `7`: ``` payu run -f -n 7 ``` This will submit subsequent jobs for the following years: 1 to 3, 4 to 6, and 7, which is a total of 3 PBS jobs. - +!!! tip + The `walltime` must be set to be long enough that the PBS job can complete. The model usually runs a single year in 90 minutes or less, but the `walltime` for a single model run is set to `2:30:00` out of an abundance of caution to make sure the model has time to run when there are occasional slower runs for unpredicatable reasons. When setting `runspersub > 1` the `walltime` doesn't need to be a simple multiple of `2:30:00` because it is highly unlikely that there will be multiple anomalously slow runs per submit. ### Running for less than one year {: id="shorter-runs"} When debugging changes to a model, it is common to reduce the run length to minimise resource consumption and return faster feedback on changes. In order to run the model for a single month, the `runtime` can be changed to @@ -361,7 +363,7 @@ When debugging changes to a model, it is common to reduce the run length to mini days: 0 ``` -With the default configuration settings, the sea ice component of {{ model }} will produce restart files only at the end of each year. If valid restart files are required when running shorter simulations, the sea ice model configuration should be modified so that restart files are produced at monthly frequencies. To do this, change the `dumpfreq = 'y'` setting to `dumpfreq = 'm'` in the `cice_in.nml` configuration file located in the `ice` control directory. +With the default configuration settings, the sea ice component of {{ model }} will produce restart files only at the end of each year. If valid restart files are required when running shorter simulations, the sea ice model configuration should be modified so that restart files are produced at monthly frequencies. To do this, change the `dumpfreq = 'y'` setting to `dumpfreq = 'm'` in the `cice_in.nml` configuration file located in the `ice` subdirectory of the control directory. ### Modify PBS resources @@ -388,15 +390,15 @@ walltime: 2:30:00 These lines can be edited to change the [PBS directives](https://opus.nci.org.au/display/Help/PBS+Directives+Explained) for the [PBS job][PBS job]. -For example, to run {{ model }} under the `ol01` project (COSIMA Working Group), uncomment the line beginning with `# project` by deleting the `#` symbol and replace `PROJECT_CODE` with `ol01`: +By default the model will be submitted to the PBS queue using your default project. To run {{ model }} using the resources of a specific project, for example the `lg87` project (ESM Working Group), uncomment the line beginning with `# project` by deleting the `#` symbol and replace `PROJECT_CODE` with `lg87`: ```yaml -project: ol01 +project: lg87 ``` !!! warning - If a project other than the user's default project is used to run {{ model }} configuration, then the `shortpath` field also needs to be uncommented and the path to the desired `/scratch/PROJECT_CODE` added.
- Doing this will make sure the same `/scratch` location is used for the _laboratory_, regardless of which project is used to run the experiment. + If more than one project is used to run an {{ model }} configuration the `shortpath` option also needs to be uncommented and the path to the desired `/scratch/PROJECT_CODE` directory added.
+ This ensures the same `/scratch` location is used for the _laboratory_, regardless of which project is used to run the experiment.

To run {{ model }}, you need to be a member of a project with allocated _Service Units (SU)_. For more information, check [how to join relevant NCI projects](/getting_started/first_steps#join-relevant-nci-projects). @@ -417,7 +419,7 @@ sync: To enable syncing, change `enable` to `True`, and set `path` to a location on `/g/data`, where _payu_ will copy output and restart folders. A sensible `path` could be: `/g/data/$PROJECT/$USER/{{model}}/experiment_name/`. !!! Warning - The {{model}} default configurations include a postprocessing [postcript](#postscripts) which converts atmospheric outputs to NetCDF format and runs in a separate PBS job. This prevents the final run's output and restart files from being automatically synced. Once the postprocessing is completed, the final outputs and restarts can be manually synced by running `payu sync` from the control directory. + The {{model}} configurations include a postprocessing [postcript](#postscripts) which converts atmospheric outputs to NetCDF format. This runs in a separate PBS job and this prevents the output and restart files of the most recent run from being automatically synced. When a series of runs completes and the final post-processing is completed run `payu sync` in the control directory to sync the final outputs and restarts. ### Saving model restarts @@ -554,7 +556,7 @@ Each submodel contains additional configuration options that are read in when th #### Collate -Rather than outputting a single diagnostic file over the whole model horizontal grid, the ocean component [MOM](/models/model_components/ocean/#modular-ocean-model-mom) typically generates diagnostic outputs as tiles, each of which spans over a portion of the model horizontal grid. +Rather than outputting a single diagnostic file over the whole model horizontal grid, the ocean component [MOM](/models/model_components/ocean/#modular-ocean-model-mom) typically generates diagnostic outputs as tiles, each of which spans a portion of model grid. The `collate` section in the `config.yaml` file controls the process that combines these smaller files into a single output file. @@ -588,7 +590,7 @@ userscripts: run: ./scripts/update_landuse_driver.sh ``` -A dictionary to run scripts or subcommands at various stages of a _payu_ submission. The above example comes from the `release-historical+concentrations` configuration, where the ```update_landuse_driver.sh``` is used to apply historical land use changes at the end of each run. +Run scripts or subcommands at various stages of a _payu_ submission. The above example comes from the `release-historical+concentrations` configuration, where the ```update_landuse_driver.sh``` is used to apply historical land use changes at the end of each run. For more information about specific `userscripts` fields, check the relevant section of [_payu_ Configuration Settings documentation](https://payu.readthedocs.io/en/latest/config.html#postprocessing). From e4fdf1528a7f2776b9ebda2558f2d2706edcf2ac Mon Sep 17 00:00:00 2001 From: Spencer Wong Date: Thu, 22 Aug 2024 13:31:49 +1000 Subject: [PATCH 10/14] Add link to release notes --- docs/models/run-a-model/run-access-esm.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/models/run-a-model/run-access-esm.md b/docs/models/run-a-model/run-access-esm.md index 9f8652db3..b25639240 100644 --- a/docs/models/run-a-model/run-access-esm.md +++ b/docs/models/run-a-model/run-access-esm.md @@ -66,8 +66,8 @@ For more information on {{ model }} configurations, check [{{model}}][model conf More information about the available experiments and the naming scheme of the branches can also be found in the [{{ model }} configs]({{ github_configs }}) GitHub repository. The first step is to choose a configuration from those available.
- -For example, if the required configuration is the co2 concentration driven pre-industrial configuration, then the branch to select is [`release-preindustrial+concentrations`](MISSING LINK). + +For example, if the required configuration is the co2 concentration driven pre-industrial configuration, then the branch to select is [`release-preindustrial+concentrations`](https://forum.access-hive.org.au/t/access-esm1-5-release-information/2352). To clone this branch to a location on _Gadi_, run: From 66d4bb64efb9c8d60ac9d4c36898928e4139817c Mon Sep 17 00:00:00 2001 From: Spencer Wong <88933912+blimlim@users.noreply.github.com> Date: Thu, 22 Aug 2024 14:46:44 +1000 Subject: [PATCH 11/14] Update sync instructions Co-authored-by: Aidan Heerdegen --- docs/models/run-a-model/run-access-esm.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/models/run-a-model/run-access-esm.md b/docs/models/run-a-model/run-access-esm.md index b25639240..050ef0663 100644 --- a/docs/models/run-a-model/run-access-esm.md +++ b/docs/models/run-a-model/run-access-esm.md @@ -416,7 +416,7 @@ sync: enable: False # set path below and change to true path: null # Set to location on /g/data or a remote server and path (rsync syntax) ``` -To enable syncing, change `enable` to `True`, and set `path` to a location on `/g/data`, where _payu_ will copy output and restart folders. A sensible `path` could be: `/g/data/$PROJECT/$USER/{{model}}/experiment_name/`. +To enable syncing, change `enable` to `True`, and set `path` to a location on `/g/data`, where _payu_ will copy output and restart folders. !!! Warning The {{model}} configurations include a postprocessing [postcript](#postscripts) which converts atmospheric outputs to NetCDF format. This runs in a separate PBS job and this prevents the output and restart files of the most recent run from being automatically synced. When a series of runs completes and the final post-processing is completed run `payu sync` in the control directory to sync the final outputs and restarts. From 30ad0d073297d25ef4c9aaaa9acfea604ec5f900 Mon Sep 17 00:00:00 2001 From: Spencer Wong Date: Fri, 23 Aug 2024 16:18:20 +1000 Subject: [PATCH 12/14] Update fields file deletion behaviour --- docs/models/run-a-model/run-access-esm.md | 16 ++++++++-------- 1 file changed, 8 insertions(+), 8 deletions(-) diff --git a/docs/models/run-a-model/run-access-esm.md b/docs/models/run-a-model/run-access-esm.md index 050ef0663..2b65577f9 100644 --- a/docs/models/run-a-model/run-access-esm.md +++ b/docs/models/run-a-model/run-access-esm.md @@ -50,9 +50,9 @@ Before running {{ model }}, you need to fulfil general prerequisites outlined in payu --version 1.1.3 - + !!! warning - _payu_ version >=1.1.3 is required + _payu_ version >=1.1.5 is required ---------------------------------------------------------------------------------------- @@ -599,21 +599,21 @@ For more information about specific `userscripts` fields, check the relevant sec Postprocessing scripts that need to be run after _payu_ has completed all steps that might alter the output directory can be run as postscripts. These run in separate PBS jobs than the main model simulation. ```yaml -postscript: -v PAYU_CURRENT_OUTPUT_DIR -P ${PROJECT} -lstorage=${PBS_NCI_STORAGE} ./scripts/NetCDF-conversion/UM_conversion_job.sh +postscript: -v PAYU_CURRENT_OUTPUT_DIR,PROJECT -lstorage=${PBS_NCI_STORAGE} ./scripts/NetCDF-conversion/UM_conversion_job.sh ``` -All {{ model }} configurations include an automatic NetCDF conversion postscript, which converts the atmosphere model's fields file format output to NetCDF in order to aid analysis and reduce storage requirements. By default, both the original atmospheric fields files, and the NetCDF files will be retained at the end of a simulation. The conversion script has an automatic deletion option, which deletes the original fields files upon successful conversion. This setting is deactivated by default, however can be enabled in the following line of the job submission script `./scripts/NetCDF-conversion/UM_conversion_job.sh`: +All {{ model }} configurations include an automatic NetCDF conversion postscript, which converts the atmosphere model's fields file format output to NetCDF in order to aid analysis and reduce storage requirements. By default, the conversion script will delete the fields files upon successful conversion, leaving only the NetCDF output. The automatic deletion can be disabled by commenting out the `--delete-ff` command line flag from the conversion job submission script `./scripts/NetCDF-conversion/UM_conversion_job.sh`. That is, changing ```bash -esm1p5_convert_nc $PAYU_CURRENT_OUTPUT_DIR +esm1p5_convert_nc $PAYU_CURRENT_OUTPUT_DIR --delete-ff ``` -To activate automatic fields file deletion, add a `--delete-ff` flag to the end of the line: +to ```bash -esm1p5_convert_nc $PAYU_CURRENT_OUTPUT_DIR --delete-ff +esm1p5_convert_nc $PAYU_CURRENT_OUTPUT_DIR # --delete-ff ``` +will prevent the output fields files from being deleted. -It's recommended to run an experiment for a few years and checking that the conversion is working as desired before activating the automatic deletion. #### Miscellaneous From 3721fbd827f8d55d01228447dc83197ae9ca8acf Mon Sep 17 00:00:00 2001 From: Spencer Wong Date: Fri, 23 Aug 2024 16:32:50 +1000 Subject: [PATCH 13/14] Small edits --- docs/models/run-a-model/run-access-esm.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/models/run-a-model/run-access-esm.md b/docs/models/run-a-model/run-access-esm.md index 2b65577f9..f2864acc0 100644 --- a/docs/models/run-a-model/run-access-esm.md +++ b/docs/models/run-a-model/run-access-esm.md @@ -419,7 +419,7 @@ sync: To enable syncing, change `enable` to `True`, and set `path` to a location on `/g/data`, where _payu_ will copy output and restart folders. !!! Warning - The {{model}} configurations include a postprocessing [postcript](#postscripts) which converts atmospheric outputs to NetCDF format. This runs in a separate PBS job and this prevents the output and restart files of the most recent run from being automatically synced. When a series of runs completes and the final post-processing is completed run `payu sync` in the control directory to sync the final outputs and restarts. + The {{model}} configurations include a postprocessing [postcript](#postscripts) which converts atmospheric outputs to NetCDF format. This runs in a separate PBS job and this prevents the output and restart files of the most recent run from being automatically synced. When a series of runs completes and the final post-processing is completed, run `payu sync` in the control directory to sync the final outputs and restarts. ### Saving model restarts @@ -601,7 +601,7 @@ Postprocessing scripts that need to be run after _payu_ has completed all steps ```yaml postscript: -v PAYU_CURRENT_OUTPUT_DIR,PROJECT -lstorage=${PBS_NCI_STORAGE} ./scripts/NetCDF-conversion/UM_conversion_job.sh ``` -All {{ model }} configurations include an automatic NetCDF conversion postscript, which converts the atmosphere model's fields file format output to NetCDF in order to aid analysis and reduce storage requirements. By default, the conversion script will delete the fields files upon successful conversion, leaving only the NetCDF output. The automatic deletion can be disabled by commenting out the `--delete-ff` command line flag from the conversion job submission script `./scripts/NetCDF-conversion/UM_conversion_job.sh`. That is, changing +All {{ model }} configurations include the above NetCDF conversion postscript, which converts the atmosphere model's fields file format output to NetCDF in order to aid analysis and reduce storage requirements. By default, the conversion script will delete the fields files upon successful completion, leaving only the NetCDF output. This automatic deletion can be disabled by commenting out the `--delete-ff` command line flag from the conversion job submission script `./scripts/NetCDF-conversion/UM_conversion_job.sh`. That is, changing ```bash esm1p5_convert_nc $PAYU_CURRENT_OUTPUT_DIR --delete-ff From 69a41070247eeb6319330523e313a1d6536c7a0c Mon Sep 17 00:00:00 2001 From: Davide Marchegiani Date: Mon, 26 Aug 2024 12:00:13 +1000 Subject: [PATCH 14/14] Minor changes to text for better readability/correctness. --- docs/models/run-a-model/run-access-esm.md | 31 ++++++++++++----------- 1 file changed, 16 insertions(+), 15 deletions(-) diff --git a/docs/models/run-a-model/run-access-esm.md b/docs/models/run-a-model/run-access-esm.md index f2864acc0..6b123f7c4 100644 --- a/docs/models/run-a-model/run-access-esm.md +++ b/docs/models/run-a-model/run-access-esm.md @@ -48,7 +48,7 @@ Before running {{ model }}, you need to fulfil general prerequisites outlined in payu --version - 1.1.3 + 1.1.5 !!! warning @@ -308,7 +308,7 @@ One of the most common changes is to adjust the duration of the model run.
{ The length of an {{model}} run is controlled by the `runtime` settings in the `config.yaml` file: -``` +```yml runtime: years: 1 months: 0 @@ -356,14 +356,14 @@ Now some practical examples: ### Running for less than one year {: id="shorter-runs"} When debugging changes to a model, it is common to reduce the run length to minimise resource consumption and return faster feedback on changes. In order to run the model for a single month, the `runtime` can be changed to -``` +```yml runtime: years: 0 months: 1 days: 0 ``` -With the default configuration settings, the sea ice component of {{ model }} will produce restart files only at the end of each year. If valid restart files are required when running shorter simulations, the sea ice model configuration should be modified so that restart files are produced at monthly frequencies. To do this, change the `dumpfreq = 'y'` setting to `dumpfreq = 'm'` in the `cice_in.nml` configuration file located in the `ice` subdirectory of the control directory. +With the default configuration settings, the sea ice component of {{ model }} will produce restart files only at the end of each year. If valid restart files are required when running shorter simulations, the sea ice model configuration should be modified so that restart files are produced at monthly frequencies. To do this, change the `dumpfreq = 'y'` setting to `dumpfreq = 'm'` in the `cice_in.nml` configuration file located in the `ice` subdirectory of the _control_ directory. ### Modify PBS resources @@ -419,7 +419,8 @@ sync: To enable syncing, change `enable` to `True`, and set `path` to a location on `/g/data`, where _payu_ will copy output and restart folders. !!! Warning - The {{model}} configurations include a postprocessing [postcript](#postscripts) which converts atmospheric outputs to NetCDF format. This runs in a separate PBS job and this prevents the output and restart files of the most recent run from being automatically synced. When a series of runs completes and the final post-processing is completed, run `payu sync` in the control directory to sync the final outputs and restarts. + The {{model}} configurations include a [postprocessing script](#postscripts) which converts atmospheric outputs to NetCDF format. This script runs in a separate PBS job and prevents the output and restart files of the most recent run from being automatically synced.
+ After a series of runs and the final post-processing is completed, manually execute `payu sync` in the _control_ directory to sync the final output and restart files. ### Saving model restarts @@ -595,13 +596,15 @@ Run scripts or subcommands at various stages of a _payu_ submission. The above e For more information about specific `userscripts` fields, check the relevant section of [_payu_ Configuration Settings documentation](https://payu.readthedocs.io/en/latest/config.html#postprocessing). -#### Postscripts {: id="postscripts"} -Postprocessing scripts that need to be run after _payu_ has completed all steps that might alter the output directory can be run as postscripts. These run in separate PBS jobs than the main model simulation. +#### Postscripts +Postprocessing scripts that run after _payu_ has completed all steps. Scripts that might alter the output directory, for example, can be run as postscripts. These run in PBS jobs separate from the main model simulation. ```yaml postscript: -v PAYU_CURRENT_OUTPUT_DIR,PROJECT -lstorage=${PBS_NCI_STORAGE} ./scripts/NetCDF-conversion/UM_conversion_job.sh ``` -All {{ model }} configurations include the above NetCDF conversion postscript, which converts the atmosphere model's fields file format output to NetCDF in order to aid analysis and reduce storage requirements. By default, the conversion script will delete the fields files upon successful completion, leaving only the NetCDF output. This automatic deletion can be disabled by commenting out the `--delete-ff` command line flag from the conversion job submission script `./scripts/NetCDF-conversion/UM_conversion_job.sh`. That is, changing + +All {{ model }} configurations include the NetCDF conversion postscript mentioned above. This script converts the [UM](/models/model_components/atmosphere#unified-model-um)'s fields file format output to NetCDF in order to facilitate analysis and reduce storage requirements. By default, the conversion script will delete the fields files upon successful completion, leaving only the NetCDF output. This automatic deletion can be disabled by commenting out the `--delete-ff` command line flag from the conversion job submission script located in the _control_ directory under `scripts/NetCDF-conversion/UM_conversion_job.sh`.
+That means changing ```bash esm1p5_convert_nc $PAYU_CURRENT_OUTPUT_DIR --delete-ff @@ -612,8 +615,6 @@ to ```bash esm1p5_convert_nc $PAYU_CURRENT_OUTPUT_DIR # --delete-ff ``` -will prevent the output fields files from being deleted. - #### Miscellaneous @@ -632,15 +633,15 @@ These configuration options are specified in files located inside a subfolder of To modify these options please refer to the User Guide of the respective model component. ### Controlling model output -Selecting the variables to save from a simulation can be a balance between enabling future analysis and minimising storage requirements. The choice and frequency of variables saved by each model can be configured from within each submodel's control directory. +Selecting the variables to save from a simulation can be a balance between enabling future analysis and minimising storage requirements. The choice and frequency of variables saved by each model can be configured from within each submodel's _control_ directory. -Each submodel's control directory contains _detailed_ and _standard_ presets for controlling the output, located in the `diagnostic_profiles` subdirectories (e.g. `~/access-esm/preindustrial+concentrations/ice/diagnostic_profiles` for the sea ice submodel). The _detailed_ profiles request a large number of variables at higher frequencies, while the _standard_ profiles restrict the output to variables more regularly used across the community. +Each submodel's _control_ directory contains _detailed_ and _standard_ presets for controlling the output, located in the `diagnostic_profiles` subdirectories (e.g. `~/access-esm/preindustrial+concentrations/ice/diagnostic_profiles` for the sea ice submodel). The _detailed_ profiles request a large number of variables at higher frequencies, while the _standard_ profiles restrict the output to variables more regularly used across the community. Selecting a preset output profile to use in a simulation can be done by pointing the following symbolic links to the desired profile: - * `STASHC` in the atmosphere control directory. - * `diag_table` in the ocean control directory. - * `ice_history.nml` in the ice control directory. + * `STASHC` in the atmosphere _control_ directory. + * `diag_table` in the ocean _control_ directory. + * `ice_history.nml` in the ice _control_ directory. For example, to select the _detailed_ output profile for the atmosphere: