diff --git a/docs/models/configurations/access-esm.md b/docs/models/configurations/access-esm.md index fbcc74dc5..2c3dad2dd 100644 --- a/docs/models/configurations/access-esm.md +++ b/docs/models/configurations/access-esm.md @@ -9,6 +9,16 @@ This means it can simulate both the physical climate and global biogeochemical c [ACCESS-ESM1.5](https://www.publish.csiro.au/es/ES19035) [@Ziehn2020] is a fully-coupled climate model with land and ocean carbon cycle components. ACCESS-ESM1.5 was developed primarily to enable Australia to participate in the [Coupled Model Intercomparison Project Phase 6 (CMIP6)](https://wcrp-cmip.org/cmip6/) with an Earth System Model (ESM) version. +ACCESS-NRI has released ACCESS-ESM1.5 configurations as an adaptation of those originally developed by [CSIRO](https://www.csiro.au/en/research/environmental-impacts/climate-change/climate-science-centre) and [CLEX CMS](https://github.com/coecms/access-esm). + +There are currently two supported configurations: + +- ***Pre-industrial concentration driven***:
+ A global coupled model configuration running in CO~2~ concentration driven mode under pre-industrial forcings, as described in [Ziehn et al. (2020)](https://doi.org/10.1071/ES19035). + Pre-industrial forcing data including atmospheric CO~2~ concentrations are primarily sourced from UKMO versions of CMIP6 inputs, with additional atmospheric forcings sourced from CMIP5 and land cover data adapted from [Lawrence et al. (2012)](https://doi.org/10.1175/JCLI-D-11-00256.1). +- ***Historical concentration driven***:
+ A global coupled model configuration running in co2 concentration driven mode under time varying historical (1850-2014) forcings, as described in [Ziehn et al. (2020)](https://doi.org/10.1071/ES19035). + Historical forcing data including atmospheric CO~2~ concentrations are primarily sourced from UKMO versions of CMIP6 inputs, with land use change data adapted from the Land-Use Harmonisation 2 (LUH2) dataset developed for CMIP6 [(Hurtt et al. 2017)](https://doi.org/10.22033/ESGF/input4MIPs.1127). ### Model components - **Atmoshpere**: [UM7.3](/models/model_components/atmosphere#unified-model-um), GA7.1 science configuration.
diff --git a/docs/models/run-a-model/run-access-esm.md b/docs/models/run-a-model/run-access-esm.md index 6b4c74e89..6b123f7c4 100644 --- a/docs/models/run-a-model/run-access-esm.md +++ b/docs/models/run-a-model/run-access-esm.md @@ -1,299 +1,249 @@ -{% set model = "ACCESS-ESM" %} +{% set model = "ACCESS-ESM1.5" %} +{% set github_configs = "https://github.com/ACCESS-NRI/access-esm1.5-configs" %} [PBS job]: https://opus.nci.org.au/display/Help/4.+PBS+Jobs +[gadi]: https://opus.nci.org.au/display/Help/0.+Welcome+to+Gadi#id-0.WelcometoGadi-Overview +[payu]: https://github.com/payu-org/payu +[model components]: /models/configurations/access-esm/#model-components +[model configurations]: /models/configurations/access-esm/#access-esm15 -!!! bug - There is a known issue with the {{ model }} configurations currently set up in the GitHub repository below. - {{ model }} configurations are currently being updated and an ACCESS-NRI release will come - in the following weeks. - If you encounter problems with the current guide in this page, consider asking for help on the [ACCESS-Hive Forum](https://forum.access-hive.org.au). +[:fontawesome-brands-github:{: class="twemoji icon-before-text"} {{ model }} configurations]({{github_configs}}){: class="text-card"} # Run {{ model }} +## About + +{{ model }} is a fully-coupled global climate model, combining atmoshpere, land, ocean, sea ice, ocean biogeochemistry and land biogeochemistry components. A description of the model and its components is available in the [{{ model }} overview][model configurations]. + +The instructions below outline how to run {{ model }} using ACCESS-NRI's software deployment pipeline, specifically designed to run on the [National Computating Infrastructure (NCI)](https://nci.org.au/about-us/who-we-are) supercomputer [_Gadi_][gadi]. + +If you are unsure whether {{ model }} is the right choice for your experiment, take a look at the overview of [ACCESS Models](/models). + +All {{model}} configurations are open source, licensed under [CC BY 4.0](https://creativecommons.org/licenses/by/4.0/?ref=chooser-v1")![CC icon](https://mirrors.creativecommons.org/presskit/icons/cc.svg?ref=chooser-v1){: style="height:1em;margin-left:0.2em;vertical-align:text-top;"}![BY icon](https://mirrors.creativecommons.org/presskit/icons/by.svg?ref=chooser-v1){: style="height:1em;margin-left:0.2em;vertical-align:text-top;"} and available on [ACCESS-NRI GitHub]({{github_configs}}). + +{{ model }} release notes are [available on the ACCESS-Hive Forum ](https://forum.access-hive.org.au/t/access-esm1-5-release-information/2352) and are updated when new releases are made available. + ## Prerequisites ### General prerequisites Before running {{ model }}, you need to fulfil general prerequisites outlined in the [First Steps](/getting_started/first_steps) section. -If you are unsure whether {{ model }} is the right choice for your experiment, take a look at the overview of [ACCESS Models](/models). - ### Model-specific prerequisites -{{ model }} is installed on NCI's supercomputer _Gadi_ and uses [_payu_](#payu), a tool for running and managing model experiments. Following these prerequisites ensures you have access to this infrastructure. - -- **Join the _access_ and _hh5_ projects at NCI**
- To join these projects, request membership on the respective [access](https://my.nci.org.au/mancini/project/access/join) and [hh5](https://my.nci.org.au/mancini/project/hh5/join) NCI project pages.
+- **Join the _vk83_ project at NCI**
+ To join this project, request membership on the [vk83](https://my.nci.org.au/mancini/project/vk83/join) NCI project page.
For more information on joining specific NCI projects, refer to [How to connect to a project](https://opus.nci.org.au/display/Help/How+to+connect+to+a+project). + - **Payu**
- _Payu_ on _Gadi_ is available through the `conda/analysis3` environment in the _hh5_ project.
- After obtaining _hh5_ project membership, load the `conda/analysis3` environment to automatically retrieve _payu_ as follows: - ``` - module use /g/data/hh5/public/modules - module load conda/analysis3 - ``` + [_Payu_][payu] is a workflow management tool for running numerical models in supercomputing environments, for which there is extensive [documentation](https://payu.readthedocs.io/en/latest/).
+ _Payu_ on _Gadi_ is available through a dedicated `conda` environment in the _vk83_ project.
+ After joining the _vk83_ project, load the `payu` module: + + module use /g/data/vk83/modules + module load payu + To check that _payu_ is available, run: - ``` - payu --version - ``` + + payu --version + payu --version - 1.0.19 + 1.1.5 + !!! warning + _payu_ version >=1.1.5 is required + +---------------------------------------------------------------------------------------- + ## Get {{ model }} configuration -{{ model }} configurations are available in the [coecms GitHub](https://github.com/coecms/access-esm) repository.
-To get it on _Gadi_, clone the {{ model }} GitHub repository by running: -``` -git clone https://github.com/coecms/access-esm.git -``` -This will create the `access-esm` folder. +All released {{ model }} configurations are available from the [{{ model }} configs]({{ github_configs }}) GitHub repository.
+Released configurations are tested and supported by ACCESS-NRI, as an adaptation of those originally developed by [CSIRO](https://www.csiro.au/en/research/environmental-impacts/climate-change/climate-science-centre) and [CLEX CMS](https://github.com/coecms/access-esm). + +For more information on {{ model }} configurations, check [{{model}}][model configurations] page. + +More information about the available experiments and the naming scheme of the branches can also be found in the [{{ model }} configs]({{ github_configs }}) GitHub repository. + +The first step is to choose a configuration from those available.
+ +For example, if the required configuration is the co2 concentration driven pre-industrial configuration, then the branch to select is [`release-preindustrial+concentrations`](https://forum.access-hive.org.au/t/access-esm1-5-release-information/2352). + +To clone this branch to a location on _Gadi_, run: + + mkdir -p ~/access-esm1.5 + cd ~/access-esm1.5 + payu clone -b expt -B release-preindustrial+concentrations {{ github_configs }} preindustrial+concentrations + +In the example above the `payu clone` command clones the concentration driven pre-industrial configuration (`-B release-preindustrial+concentrations`) +to a new experiment branch (`-b expt`) to a directory named `preindustrial+concentrations`. +!!! admonition tip + Anyone using a configuration is advised to clone only a single branch (as shown in the example above) and not the entire repository. + - git clone https://github.com/coecms/access-esm.git - Cloning into 'access-esm'... - remote: Enumerating objects: 1625, done. - remote: Counting objects: 100% (1625/1625), done. - remote: Compressing objects: 100% (575/575), done. - remote: Total 1625 (delta 1042), reused 1621 (delta 1040), pack-reused 0 - Receiving objects: 100% (1625/1625), 2.79 MiB | 11.24 MiB/s, done. - Resolving deltas: 100% (1042/1042), done. - ls ~/access-esm - atmosphere config.yaml coupler ice manifests ocean README.md + mkdir -p ~/access-esm1.5 + cd ~/access-esm1.5 + payu clone -b expt -B release-preindustrial+concentrations {{ github_configs }} preindustrial+concentrations + Cloned repository from {{ github_configs }} to directory: .../access-esm1.5/preindustrial+concentrations + Created and checked out new branch: expt + laboratory path: /scratch/.../access-esm + binary path: /scratch/.../access-esm/bin + input path: /scratch/.../access-esm/input + work path: /scratch/.../access-esm/work + archive path: /scratch/.../access-esm/archive + Updated metadata. Experiment UUID: 0635396b-678d-45f9-b81e-abd25a2d7bf0 + Added archive symlink to /scratch/.../access-esm/archive/preindustrial+concentrations-expt-0635396b + To change directory to control directory run: + cd preindustrial+concentrations -!!! warning - Some modules may interfere with `git` commands (e.g., matlab/R2018a). If you have trouble cloning the repository, run the following command before trying again: - ``` - module purge - ``` - After this step, don't forget to reload the `conda/analysis3` module to retrieve _payu_, as specified in the [Model-specific prerequisites](#model-specific-prerequisites) section. +!!! tip + _payu_ uses branches to differentiate between different experiments in the same local git repository.
+ For this reason, it is recommended to always set the cloned branch name (`expt` in the example above) to something meaningful for the planned experiment.
+ For more information refer to this [payu tutorial](https://forum.access-hive.org.au/t/access-om2-payu-tutorial/1750#select-experiment-12). + +---------------------------------------------------------------------------------------- + +## Run {{ model }} configuration +If you want to modify your configuration, refer to [Edit {{ model }} configuration](#edit-{{ model.lower() }}-configuration). + +{{ model }} configurations run on [_Gadi_][gadi] through a [PBS job][PBS job] submission managed by [_payu_][payu]. + +The general layout of a _payu_-supported model run consists of two main directories: + +- The _control_ directory contains the model configuration and serves as the execution directory for running the model (in this example, the cloned directory `~/access-esm1.5/preindustrial+concentrations`). +- The _laboratory_ directory, where all the model components reside. For {{ model }}, it is typically `/scratch/$PROJECT/$USER/access-esm`. + +This separates the small text configuration files from the larger binary outputs and inputs. In this way, the _control_ directory can be in the `$HOME` directory (as it is the only filesystem actively backed-up on _Gadi_). The quotas for `$HOME` are low and strict, which limits what can be stored there, so it is not suitable for larger files. + +The _laboratory_ directory is a shared space for all _payu_ experiments using the same model.
+Inside the _laboratory_ directory there are two subdirectories: + +- `work` → a directory where _payu_ automatically creates a temporary subdirectory while the model is run. The temporary subdirectory gets created as part of a run and then removed after the run succeeds. +- `archive` → the directory where the output is stored following each successful run. + +Within each of the above directories _payu_ automatically creates subdirectories uniquely named according to the experiment being run.
+_Payu_ also creates symbolic links in the _control_ directory pointing to the `archive` and `work` directories. + +This design allows multiple self-resubmitting experiments that share common executables and input data to be run simultaneously. + +!!! admonition warning + Files on the `/scratch` drive, such as the _laboratory_ directory, might get deleted if not accessed for several days and the `/scratch` drive is limited in space. For these reasons, all model runs which are to be kept should be moved to `/g/data/` by enabling the _sync_ step in _payu_. To know more refer to [Syncing output data](#syncing-output-data). + +### Run configuration + +To run {{ model }} configuration execute the following command from within the *control* directory: + + payu run + +This will submit a single job to the queue with a run length given by [`runtime`](#runtime) in the `config.yaml` file.
-Each {{ model }} configuration is stored as a specially-named branch in the {{ model }} GitHub repository. -To check all the available branches on the repo, run the following command inside the newly-created `access-esm` folder: -``` -git branch -a -``` - cd ~/access-esm - git branch -a - ٭ main -   remotes/origin/82ka -   remotes/origin/HEAD → origin/main -   remotes/origin/ccarouge-patch-1 -   remotes/origin/historical -   remotes/origin/last-interglacial -   remotes/origin/last-millenium -   remotes/origin/last-millenium-detailed -   remotes/origin/main -   remotes/origin/mid-holocene -   remotes/origin/pre-industrial -   remotes/origin/ssp585 + cd ~/access-esm1.5/preindustrial+concentrations + payu run + Loading input manifest: manifests/input.yaml + Loading restart manifest: manifests/restart.yaml + Loading exe manifest: manifests/exe.yaml + payu: Found modules in /opt/Modules/v4.3.0 + + qsub -q normal -P tm70 -l walltime=9000 -l ncpus=384 -l mem=1536GB -N pre-industrial -l wd -j n -v PAYU_PATH=/g/data/vk83/apps/payu/1.1.4/bin,MODULESHOME=/opt/Modules/v4.3.0,MODULES_CMD=/opt/Modules/v4.3.0/libexec/modulecmd.tcl,MODULEPATH=/g/data/vk83/modules:/etc/scl/modulefiles:/apps/Modules/restricted-modulefiles/matlab_monash:/opt/Modules/modulefiles:/opt/Modules/v4.3.0/modulefiles:/apps/Modules/modulefiles -W umask=027 -l storage=gdata/vk83 -- /g/data/vk83/apps/payu/1.1.4/bin/python3.10 /g/data/vk83/apps/payu/1.1.4/bin/payu-run + + <job-ID>.gadi-pbs> +!!! tip + You can add the `-f` option to `payu run` to let the model run even if there is an existing non-empty `work` directory, created from a previous failed run or from running `payu setup`. -The green-coloured branch (preceded by a star sign `*`) indicates the local branch you are currently in.
-The red-coloured branches are the available remote branches, formatted as `remotes/origin/`.
-To switch to a specific branch you can run the following command: -``` -git checkout -``` -For example, the pre-industrial configuration of {{ model }} is available in the `pre-industrial` branch. To use the pre-industrial configuration, run: +---------------------------------------------------------------------------------------- + +## Monitor {{ model }} runs +The `payu run` command prints out the PBS `job-ID` (formatted as `<9-digit-number>.gadi-pbs`), as the last line to the terminal.
+To print out information on the status of a specific job, you can execute the following command: ``` -git checkout pre-industrial +qstat ``` - git checkout pre-industrial - branch 'pre-industrial' set up to track 'origin/pre-industrial'. - Switched to a new branch 'pre-industrial' - git branch -   main - ٭ pre-industrial + qstat <job-ID> + Job id                Name             User             Time Use S Queue + --------------------- ---------------- ---------------- -------- - ----- + <job-ID>.gadi-pbs  pre-industrial   <$USER>            <time> R normal-exec -## Edit {{ model }} configuration - -It is good practice to create a new git branch to store all your modifications for a particular run, so as not to modify the reference configuration. - -To create a new branch called _example_run_, as a copy of the `pre-industrial` branch, from within the `access-esm` directory execute: +To show the status of all your submitted [PBS jobs][PBS job], you can execute the following command: ``` -git checkout -b example_run --no-track origin/pre-industrial +qstat -u $USER ``` -This command will also switch to the new `example_run` branch. - git branch -   main - ٭ pre-industrial - git checkout -b example_run --no-track origin/pre-industrial - Switched to a new branch 'example_run' - git branch - ٭ example_run -   main -   pre-industrial + qstat -u $USER + Job id                Name             User             Time Use S Queue + --------------------- ---------------- ---------------- -------- - ----- + <job-ID>.gadi-pbs   pre-industrial   <$USER>            <time> R normal-exec + <job-ID>.gadi-pbs   <other-job-name> <$USER>            <time> R normal-exec + <job-ID>.gadi-pbs   <other-job-name> <$USER>            <time> R normal-exec -### Payu +The default name of your job is the name of the _payu_ _control_ directory (`preindustrial+concentrations` in the example above).
+This can be overwritten by altering the `jobname` in the [PBS resources section](#modify-pbs-resources) of the `config.yaml` file. -[_Payu_](https://payu.readthedocs.io/en/latest) is a workflow management tool for running numerical models in supercomputing environments.
-The general layout of a _payu_-supported model run consists of two main directories: +_S_ indicates the status of your run, where: -- The **laboratory** directory, where all the model components reside. For {{ model }}, it is typically `/scratch/$PROJECT/$USER/access-esm`. -- The **control** directory, where the model configuration resides and from where the model is run (in this example, the cloned directory `~/access-esm`). +- _Q_ → Job waiting in the queue to start +- _R_ → Job running +- _E_ → Job ending +- _H_ → Job on hold -This distinction of directories separates the small-size configuration files from the larger binary outputs and inputs. In this way, the configuration files can be placed in the `$HOME` directory (as it is the only filesystem actively backed-up on _Gadi_), without overloading it with too much data. -Furthermore, this separation allows multiple self-resubmitting experiments that share common executables and input data to be run simultaneously. +If there are no jobs listed with your `jobname` (or if no job is listed), your run either successfully completed or was terminated due to an error.
+For more information, check [NCI documentation](https://opus.nci.org.au/display/Help/FAQ+1%3A+Why+My+Jobs+are+NOT+Running). -To set up the _laboratory_ directory, run the following command from within the _control_ directory: +### Stop a run + +If you want to manually terminate a run, you can do so by executing: ``` -payu init +qdel ``` -This creates the _laboratory_ directory, together with relevant subdirectories, depending on the configuration. The main subdirectories of interest are: - -- `work` → a temporary directory where the model is run. It gets cleaned after each run. -- `archive` → the directory where output is stored after each run. - - - cd ~/access-esm/esm-pre-industrial - payu init - laboratory path: /scratch/$PROJECT/$USER/access-esm - binary path: /scratch/$PROJECT/$USER/access-esm/bin - input path: /scratch/$PROJECT/$USER/access-esm/input - work path: /scratch/$PROJECT/$USER/access-esm/work - archive path: /scratch/$PROJECT/$USER/access-esm/archive - +which kills the specified job without waiting for it to complete. -### Edit the _Master Configuration_ file - -The `config.yaml` file located in the _control_ directory, is the _Master Configuration_ file.
-This file, which controls the general model configuration, contains several parts: - -- **PBS resources**{: id="pbs-resources"}
- ```yaml - jobname: pre-industrial - queue: normal - walltime: 3:10:00 - ``` - These lines can be edited to change the [PBS directives](https://opus.nci.org.au/display/Help/PBS+Directives+Explained) for the [PBS job].
- For example, to run {{ model }} under the `tm70` project (ACCESS-NRI), add the following line: - ``` - project: tm70 - ``` +!!! tip + If you started an {{ model }} run using the `-n` option (e.g., to [run the model for several years](#multiple-runs)), but subsequently decide not to keep running after the current process completes, you can create a file called `stop_run` in the _control_ directory.
+ This will prevent _payu_ from submitting another job. - !!! warning - The `project` entry should always refer to a project with allocated _Service Units (SU)_, that you are a member of. If not set explicitly, {{ model }} will run using your [default project](/getting_started/first_steps#change-default-project-on-gadi) (this default project still needs to have allocated SU). For more information, check [how to join relevant NCI projects](/getting_started/first_steps#join-relevant-nci-projects). +### Error and output log files -- **Link to the laboratory directory**
- ```yaml - # note: if laboratory is relative path, it is relative to /scratch/$PROJECT/$USER - laboratory: access-esm - ``` - These lines set the laboratory directory path, which is relative to `/scratch/$PROJECT/$USER`. Absolute paths can also be specified. +#### PBS output files +When the model completes a run, PBS writes the standard output and error streams to two files inside the _control_ directory: `.o` and `.e`, respectively. -- **Model**
- ```yaml - model: access - ``` - This line tells _payu_ which driver to use for the main model (`access` refers to {{ model }}). - -- **Submodels**
- - ??? code "Expand to show the full `submodels` section" - ```yaml - submodels: - - name: atmosphere - model: um - ncpus: 192 - exe: /g/data/access/payu/access-esm/bin/coe/um7.3x - input: - - /g/data/access/payu/access-esm/input/pre-industrial/atmosphere - - /g/data/access/payu/access-esm/input/pre-industrial/start_dump
- - name: ocean - model: mom - ncpus: 180 - exe: /g/data/access/payu/access-esm/bin/coe/mom5xx - input: - - /g/data/access/payu/access-esm/input/pre-industrial/ocean/common - - /g/data/access/payu/access-esm/input/pre-industrial/ocean/pre-industrial
- - name: ice - model: cice - ncpus: 12 - exe: /g/data/access/payu/access-esm/bin/coe/cicexx - input: - - /g/data/access/payu/access-esm/input/pre-industrial/ice
- - name: coupler - model: oasis - ncpus: 0 - input: - - /g/data/access/payu/access-esm/input/pre-industrial/coupler - ``` - - {{ model }} is a coupled model deploying multiple submodels (i.e. [model components](/models/configurations/access-esm/#model-components)). - This section specifies the submodels and configuration options required to execute the model correctly.
- Each submodel contains additional configuration options that are read in when the submodel is running. These options are specified in the subfolder of the _control_ directory, whose name matches the submodel's `name` (e.g., configuration options for the `atmosphere` submodel are in the `~/access-esm/esm-pre-industrial/atmosphere` directory). +These files usually contain logs about _payu_ tasks, and give an overview of the resources used by the job.
+To move these files to the `archive` directory, use the following command: +``` +payu sweep +``` -- **Collate**
- ```yaml - collate: - exe: /g/data/access/payu/access-esm/bin/mppnccombine - restart: true - mem: 4GB - ``` - The `collate` process combines a number of smaller files, which contain different parts of the model grid, into target output files. Restart files are typically tiled in the same way and will also be combined together if the `restart` option is set to `true`. +#### Model log files -- **Restart**
- ```yaml - restart: /g/data/access/payu/access-esm/restart/pre-industrial - ``` - This is the location of the files used for a warm restart. - -- **Start date and run length**{: id="runtime"}
- ```yaml - calendar: - start: - year: 101 - month: 1 - days: 1 - runtime: - years: 1 - months: 0 - days: 0 - ``` - This section specifies the start date and run length. - - !!! warning - The _run length_ (controlled by [`runtime`](#runtime)) can be different from the _total experiment length_. Also, while `runtime` can be reduced, it should not be increased to more than 1 year to avoid errors. For more information about the difference between _run length_ and _total experiment length_, or how to run the model for more than 1 year, refer to the section [Run configuration for multiple years](#run-configuration-for-multiple-years). +While the model is running, _payu_ saves the model standard output and error streams in the `access.out` and `access.err` files inside the _control_ directory, respectively.
+You can examine the contents of these files to check on the status of a run as it progresses (or after a failed run has completed). -- **Number of runs per PBS submission**{: id="runspersub"}
- ```yaml - runspersub: 1 - ``` - {{ model }} configurations are often run in multiple steps (or cycles), with _payu_ running a maximum of `runspersub` runs for every [PBS job] submission. - - !!! warning - If you increase `runspersub`, you may need to increase the `walltime` in the [PBS resources](#pbs-resources). +!!! warning + At the end of a successful run these log files are archived to the `archive` directory and will no longer be found in the _control_ directory. If they remain in the _control_ directory after the PBS job for a run has completed it means the run has failed. -To find out more about other configuration settings for the `config.yaml` file, check out [how to configure your experiment with _payu_](https://payu.readthedocs.io/en/latest/config.html). +### Model Live Diagnostics -### Edit a single {{ model }} component configuration +ACCESS-NRI developed the [Model Live Diagnostics](/model_evaluation/model_diagnostics) framework to check, monitor, visualise, and evaluate model behaviour and progress of ACCESS models currently running on _Gadi_.
+For a complete documentation on how to use this framework, check the [Model Diagnostics documentation](https://med-live-diagnostics.readthedocs.io/en/latest/index.html). -Each of [{{ model }} components](/models/configurations/access-esm/#model-components) contains additional configuration options that are read in when the model component is running.
-These options are typically useful to modify the physics used in the model, the input data, or the model variables saved in the output files. +### Trouble-shooting -These configuration options are specified in files located inside a subfolder of the _control_ directory, named according to the submodel's `name` specified in the `config.yaml` `submodels` section (e.g., configuration options for the `atmosphere` submodel are in the `~/access-esm/esm-pre-industrial/atmosphere` directory).
-To modify these options please refer to the User Guide of the respective model component. +If _payu_ doesn't run correctly for some reason, a good first step is to run the following command from within the _control_ directory: -## Run {{ model }} configuration + payu setup -After editing the configuration, you are ready to run {{ model }}.
-{{ model }} suites run on [_Gadi_](https://opus.nci.org.au/display/Help/0.+Welcome+to+Gadi#id-0.WelcometoGadi-Overview) through a [PBS job] submission managed by _payu_. +This command will: + + - create the _laboratory_ and `work` directories based on the experiment configuration + - generate manifests + - report useful information to the user, such as the location of the _laboratory_ where the `work` and `archive` directories are located -### Payu setup (optional) -As a first step, from within the _control_ directory, it is good practice to run: -``` -payu setup -``` - -This will prepare the model run, based on the experiment configuration. payu setup laboratory path: /scratch/$PROJECT/$USER/access-esm @@ -301,170 +251,413 @@ This will prepare the model run, based on the experiment configuration. input path: /scratch/$PROJECT/$USER/access-esm/input work path: /scratch/$PROJECT/$USER/access-esm/work archive path: /scratch/$PROJECT/$USER/access-esm/archive + Found experiment archive: /scratch/$PROJECT/$USER/access-esm/archive/preindustrial+concentrations-expt-0635396b + payu: Found modules in /opt/Modules/v4.3.0 + Loading access-esm1p5/2024.05.0 + Loading requirement: cice4/2024.05.21 mom5/access-esm1.5_2024.06.20 um7/2024.07.03 Loading input manifest: manifests/input.yaml Loading restart manifest: manifests/restart.yaml Loading exe manifest: manifests/exe.yaml + Making exe links Setting up atmosphere Setting up ocean Setting up ice Setting up coupler Checking exe and input manifests - Updating full hashes for 3 files in manifests/exe.yaml Creating restart manifest - Updating full hashes for 30 files in manifests/restart.yaml + Updating full hashes for 29 files in manifests/restart.yaml Writing manifests/restart.yaml - Writing manifests/exe.yaml -!!! tip - This step can be skipped as it is also included in the run command. However, running it explicitly helps to check for errors and make sure executable and restart directories are accessible. +This can help to isolate issues such as permissions problems accessing files and directories, missing files or malformed/incorrect paths. -### Run configuration +---------------------------------------------------------------------------------------- -To run {{ model }} configuration for one _run length_ (controlled by [`runtime`](#runtime) in the `config.yaml` file), from within the _control_ directory execute: +## {{ model }} outputs -``` -payu run -f -``` +At the end of a successful model run, output files, restart files and log files are moved from the `work` directory to the `archive` directory.
+Symbolic links to these directories are also provided in the _control_ directory for convenience. -This will submit a single job to the queue with a _total experiment length_ of `runtime`. If there is no previous run, it will start from the [`start`](#runtime) date indicated in the `config.yaml` file. Otherwise, it will perform a warm restart from a previously saved restart file.
+If a model run is unsuccessful, the `work` directory is left untouched to facilitate the identification of the cause of the model failure. -!!! tip - The `-f` option ensures that _payu_ will run even if there is an existing non-empty `work` directory created from a previous failed run. +Outputs and restarts are stored in subfolders within the `archive` directory, subdivided for each run of the model.
+Output and restart folders are called `outputXXX` and `restartXXX`, respectively, where _XXX_ is the run number starting from `000`. + +Model components are separated into subdirectories within the output and restart directories. - payu run -f - Loading input manifest: manifests/input.yaml - Loading restart manifest: manifests/restart.yaml - Loading exe manifest: manifests/exe.yaml - payu: Found modules in /opt/Modules/v4.3.0 - qsub -q normal -P <project> -l walltime=11400 -l ncpus=384 -l mem=1536GB -N pre-industrial -l wd -j n -v PAYU_PATH=/g/data/hh5/public/apps/miniconda3/envs/analysis3-23.01/bin,MODULESHOME=/opt/Modules/v4.3.0,MODULES_CMD=/opt/Modules/v4.3.0/libexec/modulecmd.tcl,MODULEPATH=/g/data3/hh5/public/modules:/etc/scl/modulefiles:/opt/Modules/modulefiles:/opt/Modules/v4.3.0/modulefiles:/apps/Modules/modulefiles -W umask=027 -l storage=gdata/access+gdata/hh5 -- /g/data/hh5/public/apps/miniconda3/envs/analysis3-23.01/bin/python3.9 /g/data/hh5/public/apps/miniconda3/envs/analysis3-23.01/bin/payu-run - <job-ID>.gadi-pbs + cd ~/access-esm/preindustrial+concentrations/archive + ls + metadata.yaml output000 pbs_logs restart000 -### Run configuration for multiple years +---------------------------------------------------------------------------------------- + +## Edit {{ model }} configuration + +This section describes how to modify {{ model }} configuration.
+The modifications discussed in this section can change the way {{ model }} is run by _payu_, or how its specific [model components] are configured and coupled together. -If you want to run {{ model }} configuration for multiple _run lengths_ (controlled by [`runtime`](#runtime) in the `config.yaml` file), use the option `-n`: +The `config.yaml` file located in the _control_ directory is the _Master Configuration_ file, which controls the general model configuration. It contains several parts, some of which it is more likely will need modification, and others which are rarely changed without having a deep understanding of how the model is configured. + +To find out more about configuration settings for the `config.yaml` file, refer to [how to configure your experiment with payu](https://payu.readthedocs.io/en/latest/config.html). + +### Run lengths {: id="runtime"} + +One of the most common changes is to adjust the duration of the model run.
{{model}} simulations are split into smaller _run lengths_, each with the duration specified by the `runtime` settings in the `config.yaml` file: + +The length of an {{model}} run is controlled by the `runtime` settings in the `config.yaml` file: + +```yml + runtime: + years: 1 + months: 0 + days: 0 +``` +At the end of each run length, each model component saves its state into a _restart file_, allowing the simulation to be continued in subsequent runs. + +!!! warning + The _run length_ (controlled by `runtime`) should be left at 1 year for {{model}} experiments in order to avoid errors. Shorter simulations can be useful when setting up and debugging new experiments, however they require additional configuration changes. See the section [Running for less than one year](#shorter-runs) for details. + +To run {{ model }} configuration for multiple subsequent _run lengths_ (each with duration `runtime` in the `config.yaml` file), use the option `-n` with the `payu run` command: ``` payu run -f -n ``` -This will run the configuration `number-of-runs` times with a _total experiment length_ of `runtime * number-of-runs`. The number of consecutive [PBS jobs][PBS job] submitted to the queue depends on the [`runspersub`](#runspersub) value specified in the config.yaml file. - -### Understand `runtime`, `runspersub`, and `-n` parameters +This will run the configuration `number-of-runs` times, resulting in a _total experiment length_ of `runtime * number-of-runs`. The runs will be split across a number of consecutive [PBS jobs][PBS job] submitted to the queue, as controlled by the `runspersub` value specified in the config.yaml file. + +### Understand `runtime`, `runspersub`, and `-n` parameters {: id="multiple-runs"} -With the correct use of [`runtime`](#runtime), [`runspersub`](#runspersub) and `-n` parameters, you can have full control of your run.
+It is possible to have more than one model run per queue submit. With the correct use of [`runtime`](#runtime), `runspersub` and `-n` parameters, you can have full control of your experiment.
- `runtime` defines the _run length_. - `runspersub` defines the maximum number of runs for every [PBS job] submission. +- `walltime` defines the maximum time of every [PBS job] submission. - `-n` sets the number of runs to be performed. Now some practical examples: - **Run 20 years of simulation with resubmission every 5 years**
- To have a _total experiment length_ of 20 years with a 5-year resubmission cycle, leave [`runtime`](#runtime) as the default value of `1 year` and set [`runspersub`](#runspersub) to `5`. Then, run the configuration with `-n` set to `20`: + To have a _total experiment length_ of 20 years with a 5-year resubmission cycle, leave [`runtime`](#runtime) as the default value of `1 year`, set `runspersub` to `5` and `walltime` to `10:00:00`. Then, run the configuration with `-n` set to `20`: ``` payu run-f -n 20 ``` This will submit subsequent jobs for the following years: 1 to 5, 6 to 10, 11 to 15, and 16 to 20, which is a total of 4 PBS jobs. - **Run 7 years of simulation with resubmission every 3 years**
- To have a _total experiment length_ of 7 years with a 3-year resubmission cycle, leave [`runtime`](#runtime) as the default value of `1 year` and set [`runspersub`](#runspersub) to `3`. Then, run the configuration with `-n` set to `7`: + To have a _total experiment length_ of 7 years with a 3-year resubmission cycle, leave [`runtime`](#runtime) as the default value of `1 year`, set `runspersub` to `3` and `walltime` to `6:00:00`. Then, run the configuration with `-n` set to `7`: ``` payu run -f -n 7 ``` This will submit subsequent jobs for the following years: 1 to 3, 4 to 6, and 7, which is a total of 3 PBS jobs. - -- **Run 3 months and 10 days of simulation in a single submission**
- To have a _total experiment length_ of 3 months and 10 days in a single submission, set the [`runtime`](#runtime) as follows: - ```yaml - years: 0 - months: 3 - days: 10 - ``` - Set [`runspersub`](#runspersub) to `1` (or any value > 1) and run the configuration without `-n` option (or with `-n` set to `1`): - ``` - payu run -f - ``` - -- **Run 1 year and 4 months of simulation with resubmission every 4 months**
- To have a _total experiment length_ of 1 year and 4 months (16 months), you need to split it into multiple runs. For example, 4 runs of 4 months each. In this case, set the [`runtime`](#runtime) as follows: +!!! tip + The `walltime` must be set to be long enough that the PBS job can complete. The model usually runs a single year in 90 minutes or less, but the `walltime` for a single model run is set to `2:30:00` out of an abundance of caution to make sure the model has time to run when there are occasional slower runs for unpredicatable reasons. When setting `runspersub > 1` the `walltime` doesn't need to be a simple multiple of `2:30:00` because it is highly unlikely that there will be multiple anomalously slow runs per submit. +### Running for less than one year {: id="shorter-runs"} +When debugging changes to a model, it is common to reduce the run length to minimise resource consumption and return faster feedback on changes. In order to run the model for a single month, the `runtime` can be changed to + +```yml + runtime: + years: 0 + months: 1 + days: 0 +``` + +With the default configuration settings, the sea ice component of {{ model }} will produce restart files only at the end of each year. If valid restart files are required when running shorter simulations, the sea ice model configuration should be modified so that restart files are produced at monthly frequencies. To do this, change the `dumpfreq = 'y'` setting to `dumpfreq = 'm'` in the `cice_in.nml` configuration file located in the `ice` subdirectory of the _control_ directory. + +### Modify PBS resources + +If the model has been altered and needs more time to complete, more memory, or needs to be submitted under a different NCI project, you will need to modify the following section in the `config.yaml`: + + +```yaml +# PBS configuration + +# If submitting to a different project to your default, uncomment line below +# and replace PROJECT_CODE with appropriate code. This may require setting shortpath +# project: PROJECT_CODE + +# Force payu to always find, and save, files in this scratch project directory +# shortpath: /scratch/PROJECT_CODE + +# Note: if laboratory is relative path, it is relative to shortpath/$USER +laboratory: access-esm + +jobname: pre-industrial +queue: normal +walltime: 2:30:00 +``` + +These lines can be edited to change the [PBS directives](https://opus.nci.org.au/display/Help/PBS+Directives+Explained) for the [PBS job][PBS job]. + +By default the model will be submitted to the PBS queue using your default project. To run {{ model }} using the resources of a specific project, for example the `lg87` project (ESM Working Group), uncomment the line beginning with `# project` by deleting the `#` symbol and replace `PROJECT_CODE` with `lg87`: + +```yaml +project: lg87 +``` + +!!! warning + If more than one project is used to run an {{ model }} configuration the `shortpath` option also needs to be uncommented and the path to the desired `/scratch/PROJECT_CODE` directory added.
+ This ensures the same `/scratch` location is used for the _laboratory_, regardless of which project is used to run the experiment. +

+ To run {{ model }}, you need to be a member of a project with allocated _Service Units (SU)_. For more information, check [how to join relevant NCI projects](/getting_started/first_steps#join-relevant-nci-projects). + +### Syncing output data + +The _laboratory_ directory is typically under the `/scratch` storage on _Gadi_, where [files are regularly deleted once they have been unaccessed for a period of time](https://opus.nci.org.au/pages/viewpage.action?pageId=156434436). For this reason climate model outputs need to be moved to a location with longer term storage.
+On _Gadi_, this is typically in a folder under a project code on `/g/data`. + +_Payu_ has built-in support to sync outputs, restarts and a copy of the _control_ directory git history to another location.
+This feature is controlled by the following section in the `config.yaml` file: +```yaml +# Sync options for automatically copying data from ephemeral scratch space to +# longer term storage +sync: + enable: False # set path below and change to true + path: null # Set to location on /g/data or a remote server and path (rsync syntax) +``` +To enable syncing, change `enable` to `True`, and set `path` to a location on `/g/data`, where _payu_ will copy output and restart folders. + +!!! Warning + The {{model}} configurations include a [postprocessing script](#postscripts) which converts atmospheric outputs to NetCDF format. This script runs in a separate PBS job and prevents the output and restart files of the most recent run from being automatically synced.
+ After a series of runs and the final post-processing is completed, manually execute `payu sync` in the _control_ directory to sync the final output and restart files. + +### Saving model restarts + +{{ model }} outputs restart files after every run to allow for subsequent runs to start from a previously saved model state.
+Restart files can occupy a significant amount of disk space, and keeping a lot of them is often not necessary. + +The `restart_freq` field in the `config.yaml` file specifies a strategy for retaining restart files.
+This can either be a number (in which case every _nth_ restart file is retained), or one of the following pandas-style datetime frequencies: + +- `YS` → start of the year +- `MS` → start of the month +- `D` → day +- `H` → hour +- `T` → minute +- `S` → second + +For example, to preserve the ability to restart {{ model }} every 50 model-years, set: +```yaml +restart_freq: '50YS' +``` + +The most recent sequential restarts are retained, and only deleted after a permanently archived restart file has been produced. + +For more information, check [_payu_ Configuration Settings documentation](https://payu.readthedocs.io/en/latest/config.html#model). + +### Other configuration options + +!!! warning + The following sections in the `config.yaml` file control configuration options that are rarely modified, and often require a deeper understanding of how {{ model }} is structured to be safely changed. + +#### Model configuration + +This section tells _payu_ which driver to use for the main model (`access` refers to {{ model }}). + +```yaml +model: access +``` + + +#### Submodels + +{{ model }} is a coupled model deploying multiple submodels (i.e. [model components]). + +This section specifies the submodels and configuration options required to execute {{ model }} correctly. + +Each submodel contains additional configuration options that are read in when the submodel is running. These options are specified in the subfolder of the _control_ directory, whose name matches the submodel's `name` (e.g., configuration options for the `atmosphere` submodel are in the `~/access-esm/preindustrial+concentrations/atmosphere` directory). + + +??? code "Expand to show the full `submodels` section" + ```yaml - years: 0 - months: 4 - days: 0 - ``` - Since the _run length_ is set to 4 months, set [`runspersub`](#runspersub) to `1` to resubmit your jobs every 4 months (i.e. every run). Then, run the configuration with `-n` set to `4`: - ``` - payu run -f -n 4 + submodels: + - name: atmosphere + model: um + ncpus: 192 + exe: um_hg3.exe + input: + # Aerosols + - /g/data/vk83/configurations/inputs/access-esm1p5/modern/pre-industrial/atmosphere/aerosol/global.N96/2020.05.19/OCFF_1850_ESM1.anc + - /g/data/vk83/configurations/inputs/access-esm1p5/modern/pre-industrial/atmosphere/aerosol/global.N96/2020.05.19/BC_hi_1850_ESM1.anc + - /g/data/vk83/configurations/inputs/access-esm1p5/modern/pre-industrial/atmosphere/aerosol/global.N96/2020.05.19/scycl_1850_ESM1_v4.anc + - /g/data/vk83/configurations/inputs/access-esm1p5/modern/pre-industrial/atmosphere/aerosol/global.N96/2020.05.19/Bio_1850_ESM1.anc + - /g/data/vk83/configurations/inputs/access-esm1p5/modern/share/atmosphere/aerosol/global.N96/2020.05.19/biogenic_351sm.N96L38 + - /g/data/vk83/configurations/inputs/access-esm1p5/modern/share/atmosphere/aerosol/global.N96/2020.05.19/sulpc_oxidants_N96_L38 + - /g/data/vk83/configurations/inputs/access-esm1p5/modern/share/atmosphere/aerosol/global.N96/2020.05.19/DMS_conc.N96 + # Forcing + - /g/data/vk83/configurations/inputs/access-esm1p5/modern/pre-industrial/atmosphere/forcing/global.N96/2020.05.19/ozone_1850_ESM1.anc + - /g/data/vk83/configurations/inputs/access-esm1p5/modern/pre-industrial/atmosphere/forcing/resolution_independent/2020.05.19/volcts_18502000ave.dat + # Land + - /g/data/vk83/configurations/inputs/access-esm1p5/modern/pre-industrial/atmosphere/land/biogeochemistry/global.N96/2020.05.19/Ndep_1850_ESM1.anc + - /g/data/vk83/configurations/inputs/access-esm1p5/modern/share/atmosphere/land/soiltype/global.N96/2020.05.19/qrparm.soil_igbp_vg + - /g/data/vk83/configurations/inputs/access-esm1p5/modern/share/atmosphere/land/vegetation/global.N96/2020.05.19/cable_vegfunc_N96.anc + - /g/data/vk83/configurations/inputs/access-esm1p5/share/atmosphere/land/biogeochemistry/resolution_independent/2020.05.19/modis_phenology_csiro.txt + - /g/data/vk83/configurations/inputs/access-esm1p5/share/atmosphere/land/biogeochemistry/resolution_independent/2020.05.19/pftlookup_csiro_v16_17tiles_wtlnds.csv + - /g/data/vk83/configurations/inputs/access-esm1p5/share/atmosphere/land/biogeophysics/resolution_independent/2020.05.19/def_soil_params.txt + - /g/data/vk83/configurations/inputs/access-esm1p5/share/atmosphere/land/biogeophysics/resolution_independent/2020.05.19/def_veg_params.txt + # Spectral + - /g/data/vk83/configurations/inputs/access-esm1p5/share/atmosphere/spectral/resolution_independent/2020.05.19/spec3a_sw_hadgem1_6on + - /g/data/vk83/configurations/inputs/access-esm1p5/share/atmosphere/spectral/resolution_independent/2020.05.19/spec3a_lw_hadgem1_6on + # Grids + - /g/data/vk83/configurations/inputs/access-esm1p5/modern/share/atmosphere/grids/global.N96/2020.05.19/qrparm.mask + - /g/data/vk83/configurations/inputs/access-esm1p5/share/atmosphere/grids/resolution_independent/2020.05.19/vertlevs_G3 + # STASH + - /g/data/vk83/configurations/inputs/access-esm1p5/share/atmosphere/stash/2020.05.19/ + + - name: ocean + model: mom + ncpus: 180 + exe: fms_ACCESS-CM.x + input: + # Biogeochemistry + - /g/data/vk83/configurations/inputs/access-esm1p5/modern/share/ocean/biogeochemistry/global.1deg/2020.05.19/dust.nc + - /g/data/vk83/configurations/inputs/access-esm1p5/modern/share/ocean/biogeochemistry/global.1deg/2020.05.19/ocmip2_press_monthly_om1p5_bc.nc + - /g/data/vk83/configurations/inputs/access-esm1p5/share/ocean/biogeochemistry/global.1deg/2024.07.12/bgc_param.nc + - /g/data/vk83/configurations/inputs/access-esm1p5/modern/unused/ocean/biogeochemistry/global.1deg/2020.05.19/ocmip2_fice_monthly_om1p5_bc.nc + - /g/data/vk83/configurations/inputs/access-esm1p5/modern/unused/ocean/biogeochemistry/global.1deg/2020.05.19/ocmip2_xkw_monthly_om1p5_bc.nc + # Tides + - /g/data/vk83/configurations/inputs/access-esm1p5/modern/share/ocean/tides/global.1deg/2020.05.19/roughness_amp.nc + - /g/data/vk83/configurations/inputs/access-esm1p5/modern/share/ocean/tides/global.1deg/2020.05.19/tideamp.nc + # Shortwave + - /g/data/vk83/configurations/inputs/access-esm1p5/modern/share/ocean/shortwave_penetration/global.1deg/2020.05.19/ssw_atten_depth.nc + # Grids + - /g/data/vk83/configurations/inputs/access-esm1p5/modern/share/ocean/grids/mosaic/global.1deg/2020.05.19/grid_spec.nc + + - name: ice + model: cice + ncpus: 12 + exe: cice_access_360x300_12x1_12p.exe + input: + # Grids + - /g/data/vk83/configurations/inputs/access-esm1p5/modern/share/ice/grids/global.1deg/2020.05.19/kmt.nc + - /g/data/vk83/configurations/inputs/access-esm1p5/modern/share/ice/grids/global.1deg/2020.05.19/grid.nc + # Climatology + - /g/data/vk83/configurations/inputs/access-esm1p5/modern/share/ice/climatology/global.1deg/2020.05.19/monthly_sstsss.nc + + - name: coupler + model: oasis + ncpus: 0 + input: + # Grids + - /g/data/vk83/configurations/inputs/access-esm1p5/modern/share/coupler/grids/global.oi_1deg.a_N96/2020.05.19/grids.nc + - /g/data/vk83/configurations/inputs/access-esm1p5/modern/share/coupler/grids/global.oi_1deg.a_N96/2020.05.19/areas.nc + - /g/data/vk83/configurations/inputs/access-esm1p5/modern/share/coupler/grids/global.oi_1deg.a_N96/2020.05.19/masks.nc + # Remapping weights + - /g/data/vk83/configurations/inputs/access-esm1p5/modern/share/coupler/remapping_weights/global.oi_1deg.a_N96/2020.05.19/rmp_cice_to_um1t_CONSERV_FRACNNEI.nc + - /g/data/vk83/configurations/inputs/access-esm1p5/modern/share/coupler/remapping_weights/global.oi_1deg.a_N96/2020.05.19/rmp_um1u_to_cice_CONSERV_FRACNNEI.nc + - /g/data/vk83/configurations/inputs/access-esm1p5/modern/share/coupler/remapping_weights/global.oi_1deg.a_N96/2020.05.19/rmp_um1t_to_cice_CONSERV_DESTAREA.nc + - /g/data/vk83/configurations/inputs/access-esm1p5/modern/share/coupler/remapping_weights/global.oi_1deg.a_N96/2020.05.19/rmp_cice_to_um1u_CONSERV_FRACNNEI.nc + - /g/data/vk83/configurations/inputs/access-esm1p5/modern/share/coupler/remapping_weights/global.oi_1deg.a_N96/2020.05.19/rmp_um1v_to_cice_CONSERV_FRACNNEI.nc + - /g/data/vk83/configurations/inputs/access-esm1p5/modern/share/coupler/remapping_weights/global.oi_1deg.a_N96/2020.05.19/rmp_um1t_to_cice_CONSERV_FRACNNEI.nc + - /g/data/vk83/configurations/inputs/access-esm1p5/modern/share/coupler/remapping_weights/global.oi_1deg.a_N96/2020.05.19/rmp_cice_to_um1v_CONSERV_FRACNNEI.nc + ``` -## Monitor {{ model }} runs +#### Collate -You can execute the following command to show the status of all your submitted [PBS jobs][PBS job]: +Rather than outputting a single diagnostic file over the whole model horizontal grid, the ocean component [MOM](/models/model_components/ocean/#modular-ocean-model-mom) typically generates diagnostic outputs as tiles, each of which spans a portion of model grid. +The `collate` section in the `config.yaml` file controls the process that combines these smaller files into a single output file. + +```yaml +# Collation +collate: + exe: mppnccombine.spack + restart: true + mem: 4GB + walltime: 1:00:00 + mpi: false ``` -qstat -u $USER +Restart files are typically tiled in the same way and will also be combined together if the `restart` field is set to `true`. + +#### Runlog + +```yaml +runlog: true ``` - - qstat -u $USER - Job id                Name             User             Time Use S Queue - --------------------- ---------------- ---------------- -------- - ----- - <job-ID>.gadi-pbs     pre-industrial   <$USER>            <time> R normal-exec - <job-ID-2>.gadi-pbs   <other-job-name> <$USER>            <time> R normal-exec - <job-ID-3>.gadi-pbs   <other-job-name> <$USER>            <time> R normal-exec - +When running a new configuration, _payu_ automatically commits changes with `git` if `runlog` is set to `true`. -If you changed the `jobname` in the [PBS resources](#pbs-resources) of the _Master Configuration_ file, that will appear as your job's _Name_ instead of `pre-industrial`.
+!!! warning + This should not be changed as it is an essential part of the provenance of an experiment.
+ _payu_ updates the manifest files for every run, and relies on `runlog` to save this information in the `git` history, so there is a record of all inputs, restarts, and executables used in an experiment. -_S_ indicates the status of your run, where: -- _Q_ → Job waiting in the queue to start -- _R_ → Job running -- _E_ → Job ending -- _H_ → Job on hold +#### Userscripts +```yaml +userscripts: + # Apply land use changes after each run + run: ./scripts/update_landuse_driver.sh +``` -If there are no jobs listed with your `jobname` (or if no job is listed), your run either successfully completed or was terminated due to an error.
-For more information, check [NCI documentation](https://opus.nci.org.au/display/Help/FAQ+1%3A+Why+My+Jobs+are+NOT+Running). +Run scripts or subcommands at various stages of a _payu_ submission. The above example comes from the `release-historical+concentrations` configuration, where the ```update_landuse_driver.sh``` is used to apply historical land use changes at the end of each run. -### Stop a run +For more information about specific `userscripts` fields, check the relevant section of [_payu_ Configuration Settings documentation](https://payu.readthedocs.io/en/latest/config.html#postprocessing). -If you want to manually terminate a run, you can do so by executing: + +#### Postscripts +Postprocessing scripts that run after _payu_ has completed all steps. Scripts that might alter the output directory, for example, can be run as postscripts. These run in PBS jobs separate from the main model simulation. + +```yaml +postscript: -v PAYU_CURRENT_OUTPUT_DIR,PROJECT -lstorage=${PBS_NCI_STORAGE} ./scripts/NetCDF-conversion/UM_conversion_job.sh ``` -qdel + +All {{ model }} configurations include the NetCDF conversion postscript mentioned above. This script converts the [UM](/models/model_components/atmosphere#unified-model-um)'s fields file format output to NetCDF in order to facilitate analysis and reduce storage requirements. By default, the conversion script will delete the fields files upon successful completion, leaving only the NetCDF output. This automatic deletion can be disabled by commenting out the `--delete-ff` command line flag from the conversion job submission script located in the _control_ directory under `scripts/NetCDF-conversion/UM_conversion_job.sh`.
+That means changing + +```bash +esm1p5_convert_nc $PAYU_CURRENT_OUTPUT_DIR --delete-ff ``` -### Error and output log files +to -While the model is running, _payu_ saves the standard output and standard error in the respective `access.out` and `access.err` files in the _control_ directory. You can examine the contents of these files to check on the status of a run as it progresses.
-When the model completes its run, or if it crashes, the output and error log files are by default renamed as `jobname.o` and `jobname.e`, respectively. +```bash +esm1p5_convert_nc $PAYU_CURRENT_OUTPUT_DIR # --delete-ff +``` -### Model Live Diagnostics +#### Miscellaneous -ACCESS-NRI developed the [Model Live Diagnostics](/model_evaluation/model_diagnostics) framework to check, monitor, visualise, and evaluate model behaviour and progress of ACCESS models currently running on _Gadi_.
-For a complete documentation on how to use this framework, check the [Model Diagnostics documentation](https://med-live-diagnostics.readthedocs.io/en/latest/index.html). +The following configuration settings should never require changing: -## {{ model }} outputs +```yaml +stacksize: unlimited +qsub_flags: -W umask=027 +``` -At the end of the model run, output files (and restart files) are moved from the `work` directory into the `archive` directory under `/scratch/$PROJECT/$USER/access-esm/archive/access-esm`, where they are further subdivided for each run. They are also symlinked in the _control_ directory to `~/access-esm/archive`.
-The naming format for a typical output folder is `outputXXX` and for a restart folder `restartXXX`, where _XXX_ is the run number starting from `000`. +### Edit a single {{ model }} component configuration -!!! tip - A run with a different {{ model }} configuration (different git branch) counts as a new run.
- Thus, if output folders already exist, the internal number of the new output folder will be set to the first available _XXX_ number. +Each of [{{ model }} components][model components] contains additional configuration options that are read in when the model component is running.
These options are typically useful to modify the physics used in the model, the input data, or the model variables saved in the output files. + +These configuration options are specified in files located inside a subfolder of the _control_ directory, named according to the submodel's `name` specified in the `config.yaml` `submodels` section (e.g., configuration options for the _ocean_ component are in the `~/access-esm/preindustrial+concentrations/ocean` directory).
+To modify these options please refer to the User Guide of the respective model component. -Outputs and restarts are separated in the respective folders for each model component.
-For the atmospheric output data, the files are usually [UM fieldsfile](https://code.metoffice.gov.uk/doc/um/latest/papers/umdp_F03.pdf), formatted as `a.p`. +### Controlling model output +Selecting the variables to save from a simulation can be a balance between enabling future analysis and minimising storage requirements. The choice and frequency of variables saved by each model can be configured from within each submodel's _control_ directory. + +Each submodel's _control_ directory contains _detailed_ and _standard_ presets for controlling the output, located in the `diagnostic_profiles` subdirectories (e.g. `~/access-esm/preindustrial+concentrations/ice/diagnostic_profiles` for the sea ice submodel). The _detailed_ profiles request a large number of variables at higher frequencies, while the _standard_ profiles restrict the output to variables more regularly used across the community. + +Selecting a preset output profile to use in a simulation can be done by pointing the following symbolic links to the desired profile: + + * `STASHC` in the atmosphere _control_ directory. + * `diag_table` in the ocean _control_ directory. + * `ice_history.nml` in the ice _control_ directory. + +For example, to select the _detailed_ output profile for the atmosphere: - cd /scratch/$PROJECT/$USER/access-esm/archive/esm-pre-industrial - ls - output000 pbs_logs restart000 - ls output000/atmosphere - aiihca.daa1210 aiihca.daa1810 aiihca.paa1apr aiihca.paa1jun aiihca.pea1apr aiihca.pea1jun aiihca.pga1apr aiihca.pga1jun atm.fort6.pe0 exstat ihist prefix.CNTLGEN UAFLDS_A aiihca.daa1310 aiihca.daa1910 aiihca.paa1aug aiihca.paa1mar aiihca.pea1aug aiihca.pea1mar aiihca.pga1aug aiihca.pga1mar cable.nml fort.57 INITHIS prefix.PRESM_A um_env.py aiihca.daa1410 aiihca.daa1a10 aiihca.paa1dec aiihca.paa1may aiihca.pea1dec aiihca.pea1may aiihca.pga1dec aiihca.pga1may CNTLALL ftxx input_atm.nml SIZES xhist aiihca.daa1510 aiihca.daa1b10 aiihca.paa1feb aiihca.paa1nov aiihca.pea1feb aiihca.pea1nov aiihca.pga1feb aiihca.pga1nov CONTCNTL ftxx.new namelists STASHC aiihca.daa1610 aiihca.daa1c10 aiihca.paa1jan aiihca.paa1oct aiihca.pea1jan aiihca.pea1oct aiihca.pga1jan aiihca.pga1oct debug.root.01 ftxx.vars nout.000000 thist aiihca.daa1710 aiihca.daa2110 aiihca.paa1jul aiihca.paa1sep aiihca.pea1jul aiihca.pea1sep aiihca.pga1jul aiihca.pga1sep errflag hnlist prefix.CNTLATM UAFILES_A + cd ~/access-esm/preindustrial+concentrations/atmosphere + ln -sf diagnostic_profiles/STASHC_detailed STASHC + +## Get Help + +If you have questions or need help regarding {{ model }}, consider creating a topic in the [Earth System Model category of the ACCESS Hive Forum](https://forum.access-hive.org.au/c/esm/earth-system-model/72).
+For assistance on how to request help from ACCESS-NRI, follow the [guidelines on how to get help](/about/user_support/#still-need-help). + +- [https://github.com/access-nri/access-esm1.5](https://github.com/access-nri/access-esm1.5) +- [https://opus.nci.org.au/](https://opus.nci.org.au/) - [https://github.com/coecms/esm-pre-industrial](https://github.com/coecms/esm-pre-industrial) - [https://payu.readthedocs.io/en/latest/usage.html](https://payu.readthedocs.io/en/latest/usage.html) diff --git a/docs/models/run-a-model/run-access-om.md b/docs/models/run-a-model/run-access-om.md index 07afea68f..8d79cdf45 100644 --- a/docs/models/run-a-model/run-access-om.md +++ b/docs/models/run-a-model/run-access-om.md @@ -1,24 +1,25 @@ {% set model = "ACCESS-OM2" %} -{% set access_om2_configs = "https://github.com/ACCESS-NRI/access-om2-configs" %} +{% set github_configs = "https://github.com/ACCESS-NRI/access-om2-configs" %} [cosima]: https://cosima.org.au [PBS job]: https://opus.nci.org.au/display/Help/4.+PBS+Jobs [payu]: https://github.com/payu-org/payu -[model components]: https://access-hive.org.au/models/configurations/access-om/#model-components +[model components]: /models/configurations/access-om/#model-components +[model configurations]: /models/configurations/access-om/#access-om2 [gadi]: https://opus.nci.org.au/display/Help/0.+Welcome+to+Gadi#id-0.WelcometoGadi-Overview -[:fontawesome-brands-github:{: class="twemoji icon-before-text"} {{ model }} configurations](https://github.com/ACCESS-NRI/access-om2-configs){: class="text-card"} +[:fontawesome-brands-github:{: class="twemoji icon-before-text"} {{ model }} configurations]({{github_configs}}){: class="text-card"} # Run {{ model }} ## About -{{ model }} is an Ocean Sea-Ice model. More information is available in the [{{ model }} overview](https://access-hive.org.au/models/configurations/access-om/#access-om2). +{{ model }} is an Ocean Sea-Ice model. More information is available in the [{{ model }} overview][model configurations]. The instructions below outline how to run {{ model }} using ACCESS-NRI's software deployment pipeline, specifically designed to run on the [National Computating Infrastructure (NCI)](https://nci.org.au/about-us/who-we-are) supercomputer [_Gadi_][gadi]. -If you are unsure whether {{ model }} is the right choice for your experiment, take a look at the overview of [ACCESS Models](https://access-hive.org.au/models). +If you are unsure whether {{ model }} is the right choice for your experiment, take a look at the overview of [ACCESS Models](/models). -All Model configurations are open source, licensed under [CC BY 4.0](https://creativecommons.org/licenses/by/4.0/?ref=chooser-v1")![CC icon](https://mirrors.creativecommons.org/presskit/icons/cc.svg?ref=chooser-v1){: style="height:1em;margin-left:0.2em;vertical-align:text-top;"}![BY icon](https://mirrors.creativecommons.org/presskit/icons/by.svg?ref=chooser-v1){: style="height:1em;margin-left:0.2em;vertical-align:text-top;"} and available on [ACCESS-NRI GitHub](https://github.com/ACCESS-NRI/access-om2-configs). +All {{model}} configurations are open source, licensed under [CC BY 4.0](https://creativecommons.org/licenses/by/4.0/?ref=chooser-v1")![CC icon](https://mirrors.creativecommons.org/presskit/icons/cc.svg?ref=chooser-v1){: style="height:1em;margin-left:0.2em;vertical-align:text-top;"}![BY icon](https://mirrors.creativecommons.org/presskit/icons/by.svg?ref=chooser-v1){: style="height:1em;margin-left:0.2em;vertical-align:text-top;"} and available on [ACCESS-NRI GitHub]({{github_configs}}). {{ model }} release notes are [available on the ACCESS-Hive Forum](https://forum.access-hive.org.au/t/access-om2-release-information/1602/) and are updated when new releases are made available. @@ -26,7 +27,7 @@ All Model configurations are open source, licensed under [CC BY 4.0](https://cre ### General prerequisites -Before running {{ model }}, you need to fulfil general prerequisites outlined in the [First Steps](https://access-hive.org.au/getting_started/first_steps) section. +Before running {{ model }}, you need to fulfil general prerequisites outlined in the [First Steps](/getting_started/first_steps) section. ### Model-specific prerequisites @@ -57,12 +58,12 @@ Before running {{ model }}, you need to fulfil general prerequisites outlined in ## Get {{ model }} configuration -All released {{ model }} configurations are available from the [{{ model }} configs]({{ access_om2_configs }}) GitHub repository.
+All released {{ model }} configurations are available from the [{{ model }} configs]({{ github_configs }}) GitHub repository.
Released configurations are tested and supported by ACCESS-NRI, as an adaptation of those originally developed by [COSIMA][cosima]. -For more information on {{ model }} configurations, check [ACCESS-OM2](/models/configurations/access-om#access-om2) page. +For more information on {{ model }} configurations, check [{{model}}][model configurations] page. -More information about the available experiments and the naming scheme of the branches can also be found in the [{{ model }} configs]({{ access_om2_configs }}) GitHub repository. +More information about the available experiments and the naming scheme of the branches can also be found in the [{{ model }} configs]({{ github_configs }}) GitHub repository. The first step is to choose a configuration from those available.
For example, if the required configuration is the 1° horizontal resolution with repeat-year _JRA55_ forcing (without BGC), then the branch to select is [`release-1deg_jra55_ryf`](https://github.com/ACCESS-NRI/access-om2-configs/tree/release-1deg_jra55_ryf). @@ -71,7 +72,7 @@ To clone this branch to a location on _Gadi_, run: mkdir -p ~/access-om2 cd ~/access-om2 - payu clone -b expt -B release-1deg_jra55_ryf {{ access_om2_configs }} 1deg_jra55_ryf + payu clone -b expt -B release-1deg_jra55_ryf {{ github_configs }} 1deg_jra55_ryf In the example above the `payu clone` command clones the 1° repeat-year JRA55 configuration (` -B release-1deg_jra55_ryf`) to a new experiment branch (`-b expt`) to a directory named `1deg_jra55_ryf`. @@ -155,20 +156,6 @@ For information about `restart_period`, refer to [Change run length](#change-run !!! tip You can add the `-f` option to `payu run` to let the model run even if there is an existing non-empty `work` directory, created from a previous failed run or from running `payu setup`. -### Run configuration for more than 5 years - -As mentioned in the [Change run length](#change-run-length) section, you cannot specify more than 5 years as `restart_period`.
-If you want to run a configuration for more than 5 years, you need to use the `-n` option: - - payu run -n - -This will run {{ model }} `number-of-runs` consecutive times, each with a *run length* equal to `restart_period`.
-This way, the *total experiment length* will be `restart_period * number-of-runs`. - -For example, to run a configuration for a total of 50 years with a `restart_period` of 5 years, the `number-of-runs` should be set to `10`: - - payu run -n 10 - ---------------------------------------------------------------------------------------- ## Monitor {{ model }} runs @@ -339,7 +326,21 @@ For example, to make the model run for 1 year, 4 months and 10 days, change `res While `restart_period` can be reduced, it should not be increased to more than 5 years to avoid errors.

It is also important to differentiate between _run length_ and _total experiment length_.
- For more information about their difference, or how to run the model for more than 5 years, refer to the section [Run configuration for more than 5 years](#run-configuration-for-more-than-5-years). + For more information about their difference, or how to run the model for more than 5 years, refer to [Run configuration for more than 5 years](#run-configuration-for-more-than-5-years). + +#### Run configuration for more than 5 years + +As mentioned in the [Change run length](#change-run-length) section, you cannot specify more than 5 years as `restart_period`.
+If you want to run a configuration for more than 5 years, you need to use the `-n` option: + + payu run -n + +This will run {{ model }} `number-of-runs` consecutive times, each with a *run length* equal to `restart_period`.
+This way, the *total experiment length* will be `restart_period * number-of-runs`. + +For example, to run a configuration for a total of 50 years with a `restart_period` of 5 years, the `number-of-runs` should be set to `10`: + + payu run -n 10 ### Modify PBS resources @@ -390,7 +391,7 @@ sync: - '*.nc.*' - 'iceh.????-??-??.nc' ``` -To enable syncing, change `enable` to `True`, and set `path` to a location on `/g/data`, where _payu_ will copy output and restart folders. A sensible `path` could be: `/g/data/$PROJECT/$USER/experiment_name/`. +To enable syncing, change `enable` to `True`, and set `path` to a location on `/g/data`, where _payu_ will copy output and restart folders. A sensible `path` could be: `/g/data/$PROJECT/$USER/{{model}}/experiment_name/`. !!! admonition tip The {{model}} default configurations include a [userscript](#userscripts) in the _sync_ step that concatenates daily history/diagnostic output from the Sea-Ice model (CICE5) into monthly files. This speeds up access and saves storage space, but will only run if _sync_ is enabled.