diff --git a/docs/js/miscellaneous.js b/docs/js/miscellaneous.js index 46245f300..9f25cfedb 100644 --- a/docs/js/miscellaneous.js +++ b/docs/js/miscellaneous.js @@ -62,7 +62,7 @@ function tabFunctionality() { document.querySelectorAll('[href^="#"]:not([class^="md"])').forEach(el => { let href = el.getAttribute('href'); let tabEl = document.getElementById(href.slice(1,)) - if (tabEl.parentElement.classList.contains("tabLabels")) { + if (tabEl?.parentElement.classList.contains("tabLabels")) { el.addEventListener("click",(e) => openTab(tabEl), false); } }) diff --git a/docs/models/run-a-model/run-access-cm.md b/docs/models/run-a-model/run-access-cm.md index 696572c4f..00a0fd4f8 100644 --- a/docs/models/run-a-model/run-access-cm.md +++ b/docs/models/run-a-model/run-access-cm.md @@ -1,33 +1,13 @@ {% set model = "ACCESS-CM" %} - -
- - -
- -
- -
-

Run {{ model }} from ARE / Gadi

-
- -
-

Run {{ model }} from accessdev

-
-
- - +# Run {{ model }}
- The workflow to run {{ model }} is currently in transition from accessdev to ARE/Gadi. + Important for accessdev users!
- The above tabs allow you choose the type of workflow you would prefer to follow. + If you are an accessdev user, make sure you are a member of hr22 and ki32 projects.
-
- If you are new to ACCESS-CM, we strongly recommend that you follow the ARE/Gadi workflow, as the accessdev workflow will soon be discontinued. + Then, refer to instructions on how to Set up persistent session worflow for {{ model }}, and how to port suites from accessdev.
- ## Prerequisites - ### General prerequisites Before running {{ model }}, you need to fulfil general prerequisites outlined in the First Steps section. @@ -51,262 +31,224 @@ If you are unsure whether {{ model }} is the right choice for your experiment, t For more information on how to join specific NCI projects, refer to How to connect to a project. - -
- -
-
  • - Connection to an ARE VDI Desktop -
    - To run {{ model }}, you need to be able to start an Australian Research Environment (ARE) VDI Desktop session. -
    - If you are not familiar with ARE, check out the Getting Started on ARE section. -
    • -
    - -
    -
  • - Connection to accessdev -
    - To run {{ model }}, you need to be able to connect to accessdev. This is an NCI server providing configuration and run control for {{ model }}. -
    - You also need to ensure there is correct communication between accessdev and Gadi. -
    - To complete these steps, refer to the SSH & SSH Agent section in the Getting Connected to Accessdev guide. -
    • -
    -
    - +
  • + Connection to an ARE VDI Desktop (optional) +
    + To run {{ model }}, start an Australian Research Environment (ARE) VDI Desktop session. +
    + If you are not familiar with ARE, check out the Getting Started on ARE section. +
    • -------------------------------------------- -## Setup for your chosen workflow -
    - -
    -
    - Your chosen workflow is ARE / Gadi. If you would prefer to run {{ model }} on accessdev, then select the accessdev workflow. -
    -

    Launch ARE VDI Session

    - Go to the ARE VDI page and launch a session with the following directives: - - Launch the ARE session and, once it starts, click on Launch VDI Desktop. - Launch ARE VDI session +## Set up an ARE VDI Desktop (optional) +To skip this step and instead run {{ model }} from Gadi login node, refer to instructions on how to Set up {{ model }} persistent session. + +### Launch ARE VDI Session +Go to the ARE VDI page and launch a session with the following entries: +
    - -
    -
    - Your chosen workflow is accessdev. If you would prefer to run {{ model }} on ARE / Gadi, then select the ARE / Gadi workflow. -
    - Login to accessdev by runnning: - 
    ssh accessdev
    -
    - If you have not yet set up your accessdev connection through `ssh`, please check the Getting Connected to Accessdev guide. -
    -
    + {{ model }} does not run directly on ARE. +
    + This means that the ARE VDI session only needs to carry out setup steps as well as starting the run itself. All these tasks can be done within 2 hours. + +
  • + Queue → normalbw +
    • +
  • + Compute Size → tiny (1 CPU) +
    + As mentioned above, the ARE VDI session is only needed for setup and startup tasks, which can be easily accomplished with 1 CPU. +
    • +
  • + Project → a project of which you are a member. +
    + The project must have allocated Service Units (SU) to run your simulation. Usually, but not always, this corresponds to your $PROJECT. +
    + For more information, refer to how to Join relevant NCI projects. +
    • +
  • + Storage → gdata/access+gdata/hh5+gdata/hr22+gdata/ki32 (minimum) +
    + This is a list of all project data storage (joined by + signs) needed for the {{ model }} simulation. In ARE, storage locations need to be explicitly defined to access data from within a VDI instance. +
    + Every {{ model }} simulation can be unique and input data can originate from various sources. Hence, if your simulation requires data stored in project folders other than access, hh5, hr22 or ki32, you need to add those projects to the storage path. +
    + For example, if your {{ model }} simulation requires data stored in /g/data/tm70 and /scratch/w40, your full storage path will be: gdata/access+gdata/hh5+gdata/hr22+gdata/ki32+gdata/tm70+scratch/w40 +
    • + +Launch the ARE session and, once it starts, click on Launch VDI Desktop. +Launch ARE VDI session +
    + +### Open the terminal in the VDI Desktop +Once the new tab opens, you will see a Desktop with a few folders on the left. +
    +To open the terminal, click on the black terminal icon at the top of the window. You should now be connected to a Gadi computing node. +Open ARE VDI terminal + +## Set up {{ model }} persistent session +To support the use of long-running processes, such as ACCESS model runs, NCI provides a service on Gadi called persistent sessions. + +To run {{ model }}, you need to start a persistent session and set it as the target session for the model run. + +### Start a new persistent session +To start a new persistent session on Gadi, using either a login node or an ARE terminal instance, run the following command: +
    persistent-sessions start <name>
    + +This will start a persistent session with the given name that runs under your default project. +
    +If you want to assign a different project to the persistent session, use the option -p: +
    persistent-sessions start -p <project> <name>
    + +
    + While the project assigned to a persistent session does not have to be the same as the project used to run the {{ model }} configuration, it does need to have allocated Service Units (SU). +
    + For more information, check how to Join relevant NCI projects. +
    + + persistent-sessions start <name> + session <persistent-session-uuid> running - connect using +  ssh <name>.<$USER>.<project>.ps.gadi.nci.org.au + + +To list all active persistent sessions run: +
    persistent-sessions list
    + + + persistent-sessions list +              UUID  PROJECT   ADDRESS    CPUTIME MEMORY + <persistent-session-uuid> <project> 10.9.0.62 00:00:05.213 30.5M + + + +The label of a newly-created persistent session has the following format: +
    +<name>.<$USER>.<project>.ps.gadi.nci.org.au. + +### Specify {{ model }} target persistent session + +After starting the persistent session, it is essential to assign it to the {{ model }} run. +
    +The easiest way to do this is to insert the persistent session label into the file ~/.persistent-sessions/cylc-session. +
    +You can do it manually, or by running the following command: + +
    cat > ~/.persistent-sessions/cylc-session <<< <name>.<$USER>.<project>.ps.gadi.nci.org.au
    + +For example, if the user abc123 started a persistent session named cylc under the project xy00, the command will be: + + + cat > ~/.persistent-sessions/cylc-session <<< cylc.abc123.xy00.ps.gadi.nci.org.au + cat ~/.persistent-sessions/cylc-session + cylc.abc123.xy00.ps.gadi.nci.org.au + + +For more information on how to specify the target session, refer to Specify Target Session with Cylc7 Suites. +
    + You can simultaneously submit multiple {{ model }} runs using the same persistent session without needing to start a new one. Hence, the process of specifying the target persistent session for {{ model }} should only need to be done once. +
    + After specifying the {{ model }} target persistent session the first time, to run {{ model }} you just need to make sure to have an active persistent session named like the {{ model }} target persistent session.
    - + +### Terminate a persistent session +To stop a persistent session, run: +
    persistent-sessions kill <persistent-session-uuid>
    +
    + When you terminate a persistent session, any model running on that session will stop. Therefore, you should check whether you have any active model runs before terminating a persistent session. +
    + ## Get {{ model }} suite {{ model }} comprises the model components UM, MOM, CICE, CABLE and OASIS. These components, which have different model parameters, input data and computer-related information, need to be packaged together as a suite in order to run.
    Each {{ model }} suite has a suite-ID in the format u-<suite-name>, where <suite-name> is a unique identifier.
    -
    - -
    - For this example you can use u-cy339, which is a preindustrial experiment suite. -
    - Typically, an existing suite is copied and then edited as needed for a particular run. -
    - -
    - For this example you can use u-br565, which is the CMIP6-release preindustrial experiment suite. -
    - Typically, an existing suite is copied and then edited as needed for a particular run. -
    -
    - +For this example you can use u-cy339, which is a pre-industrial experiment suite. +
    +Typically, an existing suite is copied and then edited as needed for a particular run. ### Copy {{ model }} suite with Rosie Rosie is an SVN repository wrapper with a set of options specific for {{ model }} suites.
    - -
    - -
    - To copy an existing suite on Gadi you need to follow three main steps: -
      -
    1. - Get Cylc7 setup -
      - To get the Cylc7 setup required to run {{ model }}, execute the following commands: - 
      module use /g/data/hr22/modulefiles
-module load cylc7
      - - module use /g/data/hr22/modulefiles - module load cylc7 - Loading cylc7/23.03 -  Loading requirement: mosrs-setup/1.0.1 - -
      2. -
    2. - MOSRS authentication -
      - To authenticate using your MOSRS credentials, run: - 
      mosrs-auth
      - - mosrs-auth - INFO: You need to enter your MOSRS credentials here so that GPG can cache your password. - Please enter the MOSRS password for <MOSRS-username>: - INFO: Checking your credentials using Subversion. Please wait. - INFO: Successfully accessed Subversion with your credentials. - INFO: Checking your credentials using rosie. Please wait. - INFO: Successfully accessed rosie with your credentials. - -
      3. -
    3. - Copy a suite -
      -
        -
      • - Local-only copy -
        - To create a local copy of the <suite-ID> from the UKMO repository, run: - 
        rosie checkout <suite-ID>
        - - rosie checkout <suite-ID> - [INFO] create: /home/565/<$USER>/roses - [INFO] <suite-ID>: local copy created at /home/565/<$USER>/roses/<suite-ID> - - This option is mostly used for testing and examining existing suites. -
        • -
      • - Remote and local copy -
        - Alternatively, to create a new copy of an existing <suite-ID> both locally and remotely in the UKMO repository, run: - 
        rosie copy <suite-ID>
        - - rosie copy <suite-ID> - Copy "<suite-ID>/trunk@<trunk-ID>" to "u-?????"? [y or n (default)] y - [INFO] <new-suite-ID>: created at https://code.metoffice.gov.uk/svn/roses-u/<suite-n/a/m/e/> - [INFO] <new-suite-ID>: copied items from <suite-ID>/trunk@<trunk-ID> - [INFO] <suite-ID>: local copy created at /home/565/<$USER>/roses/<new-suite-ID> - - When a new suite is created in this way, a unique <suite-ID> is generated within the repository and populated with descriptive information about the suite and its initial configuration. -
        • -
      -
      4. -
    - For additional rosie options, run: - 
    rosie help
    +To copy an existing suite on Gadi you need to follow three main steps: +
      +
    1. + Get Cylc7 setup
      - Suites are created in the user's home directory on Gadi under ~/roses/<suite-ID>. + To get the Cylc7 setup required to run {{ model }}, execute the following commands: + 
      module use /g/data/hr22/modulefiles
+module load cylc7/23.09
      + + module use /g/data/hr22/modulefiles + module load cylc7/23.09 + Using the cylc session <name>.<$USER>.<project>.ps.gadi.nci.org.au + + Loading cylc7/23.09 +  Loading requirement: mosrs-setup/1.0.1 + +
      2. +
      + Make sure to load Cylc version 23.09 (or later), as earlier versions do not support the persistent sessions workflow. +
      + Also, before loading the Cylc module, make sure to have started a persistent session and assigned it to the {{ model }} workflow. For more information about these steps, refer to instructions on how to Set up {{ model }} persistent session.
      - -
      - To copy an existing suite on accessdev you need to follow two main steps: -
        +
      1. + MOSRS authentication +
        + To authenticate using your MOSRS credentials, run: + 
        mosrs-auth
        + + mosrs-auth + INFO: You need to enter your MOSRS credentials here so that GPG can cache your password. + Please enter the MOSRS password for <MOSRS-username>: + INFO: Checking your credentials using Subversion. Please wait. + INFO: Successfully accessed Subversion with your credentials. + INFO: Checking your credentials using rosie. Please wait. + INFO: Successfully accessed rosie with your credentials. + +
        2. +
      2. + Copy a suite +
        +
        • - MOSRS authentication + Local-only copy
          - To authenticate using your MOSRS credentials, run: - 
          mosrs-auth
          + To create a local copy of the <suite-ID> from the UKMO repository, run: + 
          rosie checkout <suite-ID>
          - mosrs-auth - Please enter the MOSRS password for <MOSRS-username>: - Successfully authenticated with MOSRS as <MOSRS-username> + rosie checkout <suite-ID> + [INFO] create: /home/565/<$USER>/roses + [INFO] <suite-ID>: local copy created at /home/565/<$USER>/roses/<suite-ID> + This option is mostly used for testing and examining existing suites.
        • - Copy a suite -
          -
            -
          • - Local-only copy -
            - To create a local copy of the <suite-ID> from the UKMO repository, run: - 
            rosie checkout <suite-ID>
            - - rosie checkout <suite-ID> - [INFO] create: /home/565/<$USER>/roses - [INFO] <suite-ID>: local copy created at /home/565/<$USER>/roses/<suite-ID> - - This option is mostly used for testing and examining existing suites. -
            • -
          • - Remote and local copy -
            - Alternatively, to create a new copy of an existing <suite-ID> both locally and remotely in the UKMO repository, run: - 
            rosie copy <suite-ID>
            - - rosie copy <suite-ID> - Copy "<suite-ID>/trunk@<trunk-ID>" to "u-?????"? [y or n (default)] y - [INFO] <new-suite-ID>: created at https://code.metoffice.gov.uk/svn/roses-u/<suite-n/a/m/e/> - [INFO] <new-suite-ID>: copied items from <suite-ID>/trunk@<trunk-ID> - [INFO] <suite-ID>: local copy created at /home/565/<$USER>/roses/<new-suite-ID> - - When a new suite is created in this way, a unique <suite-ID> is generated within the repository and populated with descriptive information about the suite and its initial configuration. -
            • -
          + Remote and local copy +
          + Alternatively, to create a new copy of an existing <suite-ID> both locally and remotely in the UKMO repository, run: + 
          rosie copy <suite-ID>
          + + rosie copy <suite-ID> + Copy "<suite-ID>/trunk@<trunk-ID>" to "u-?????"? [y or n (default)] y + [INFO] <new-suite-ID>: created at https://code.metoffice.gov.uk/svn/roses-u/<suite-n/a/m/e/> + [INFO] <new-suite-ID>: copied items from <suite-ID>/trunk@<trunk-ID> + [INFO] <suite-ID>: local copy created at /home/565/<$USER>/roses/<new-suite-ID> + + When a new suite is created in this way, a unique <suite-ID> is generated within the repository and populated with descriptive information about the suite and its initial configuration.
          • -
      - For additional rosie options, run: - 
      rosie help
      -
      - Suites are created in the user's home directory on accessdev under ~/roses/<suite-ID>. -
      -
    - + + + +For additional rosie options, run: +
    rosie help
    +
    +Suites are created in the user's home directory on Gadi under ~/roses/<suite-ID>. Each suite directory usually contains two subdirectories and three files: For the atmospheric output data, the files are typically a UM fieldsfile or netCDF file, formatted as <suite-name>a.p<output-stream-identifier><year><month-string>. -
    -
    - -
    - For the u-cy339 suite in this example, the atm directory contains: - - cd /scratch/<$PROJECT>/<$USER>/archive - ls - cy339 <other-suite-name> <other-suite-name> - cd cy339 - ls - history restart - ls history/atm - cy339a.pd0950apr.nc cy339a.pd0950aug.nc cy339a.pd0950dec.nc cy339a.pd0950feb.nc cy339a.pd0950jan.nc cy339a.pd0950jul.nc cy339a.pd0950jun.nc cy339a.pd0950mar.nc cy339a.pd0950may.nc cy339a.pd0950nov.nc cy339a.pd0950oct.nc cy339a.pd0950sep.nc cy339a.pd0951apr.nc cy339a.pd0951aug.nc cy339a.pd0951dec.nc cy339a.pm0950apr.nc cy339a.pm0950aug.nc cy339a.pm0950dec.nc cy339a.pm0950feb.nc cy339a.pm0950jan.nc cy339a.pm0950jul.nc cy339a.pm0950jun.nc cy339a.pm0950mar.nc cy339a.pm0950may.nc cy339a.pm0950nov.nc cy339a.pm0950oct.nc cy339a.pm0950sep.nc cy339a.pm0951apr.nc cy339a.pm0951aug.nc cy339a.pm0951dec.nc netCDF - -
    - -
    - For the u-br565 suite in this example, the atm directory contains: - - cd /scratch/<$PROJECT>/<$USER>/archive - ls - br565 <other-suite-name> <other-suite-name> - cd br565 - ls - history restart - ls history/atm - br565a.pd0950apr.nc br565a.pd0950aug.nc br565a.pd0950dec.nc br565a.pd0950feb.nc br565a.pd0950jan.nc br565a.pd0950jul.nc br565a.pd0950jun.nc br565a.pd0950mar.nc br565a.pd0950may.nc br565a.pd0950nov.nc br565a.pd0950oct.nc br565a.pd0950sep.nc br565a.pd0951apr.nc br565a.pd0951aug.nc br565a.pd0951dec.nc br565a.pm0950apr.nc br565a.pm0950aug.nc br565a.pm0950dec.nc br565a.pm0950feb.nc br565a.pm0950jan.nc br565a.pm0950jul.nc br565a.pm0950jun.nc br565a.pm0950mar.nc br565a.pm0950may.nc br565a.pm0950nov.nc br565a.pm0950oct.nc br565a.pm0950sep.nc br565a.pm0951apr.nc br565a.pm0951aug.nc br565a.pm0951dec.nc netCDF - -
    -
    - + +For the u-cy339 suite in this example, the atm directory contains: + + cd /scratch/<$PROJECT>/<$USER>/archive + ls + cy339 <other-suite-name> <other-suite-name> + cd cy339 + ls + history restart + ls history/atm + cy339a.pd0950apr.nc cy339a.pd0950aug.nc cy339a.pd0950dec.nc cy339a.pd0950feb.nc cy339a.pd0950jan.nc cy339a.pd0950jul.nc cy339a.pd0950jun.nc cy339a.pd0950mar.nc cy339a.pd0950may.nc cy339a.pd0950nov.nc cy339a.pd0950oct.nc cy339a.pd0950sep.nc cy339a.pd0951apr.nc cy339a.pd0951aug.nc cy339a.pd0951dec.nc cy339a.pm0950apr.nc cy339a.pm0950aug.nc cy339a.pm0950dec.nc cy339a.pm0950feb.nc cy339a.pm0950jan.nc cy339a.pm0950jul.nc cy339a.pm0950jun.nc cy339a.pm0950mar.nc cy339a.pm0950may.nc cy339a.pm0950nov.nc cy339a.pm0950oct.nc cy339a.pm0950sep.nc cy339a.pm0951apr.nc cy339a.pm0951aug.nc cy339a.pm0951dec.nc netCDF + ### Restart files The restart files can be found in the /scratch/$PROJECT/$USER/archive/<suite-name>/restart directory, where they are categorised according to model components (similar to the history folder above).
    The atmospheric restart files, which are UM fieldsfiles, are formatted as <suite-name>a.da<year><month><day>_00. -
    - -
    - For the u-cy339 suite in this example, the atm directory contains: - - ls /scratch/<$PROJECT>/<$USER>/archive/cy339/restart/atm - cy339a.da09500201_00 cy339a.da09510101_00 cy339.xhist-09500131 cy339.xhist-09501231 - -
    - -
    - For the u-br565 suite in this example, the atm directory contains: - - ls /scratch/<$PROJECT>/<$USER>/archive/br565/restart/atm - br565a.da09500201_00 br565a.da09510101_00 br565.xhist-09500131 br565.xhist-09501231 - -
    -
    - + +For the u-cy339 suite in this example, the atm directory contains: + + ls /scratch/<$PROJECT>/<$USER>/archive/cy339/restart/atm + cy339a.da09500201_00 cy339a.da09510101_00 cy339.xhist-09500131 cy339.xhist-09501231 + + Files formatted as <suite-name>a.xhist-<year><month><day> contain metadata information. +## Port suites from accessdev +accessdev was the server used for {{ model }} run submission workflow before the update to persistent sessions. +
    +If you have a suite that was running on accessdev, you can run it using persistent sessions by carrying out the following steps: + +
      +
    1. + Initialisation step +
      + To set the correct SSH configuration for Cylc, some SSH keys need to be created in the ~/.ssh directory. +
      + To create the required SSH keys, run the following command: + 
      /g/data/hr22/bin/gadi-cylc-setup-ps -y
      +
      + You only need to run this initialisation step once. +
      +
      2. +
    2. + Set host to localhost +
      + To enable Cylc to submit PBS jobs directly from the persistent session, the suite configuration should have its host set as localhost. +
      + You can manually set all occurrences of host to localhost in the suite configuration files. +
      + Alternatively, you can run the following command in the suite folder: + 
      grep -rl --exclude-dir=.svn "host\s*=" . | xargs sed -i 's/\(host\s*=\s*\).*/\1localhost/g'
      +
      3. +
    3. + Add gdata/hr22 and gdata/ki32 in the PBS storage directives +
      + As the persistent sessions workflow uses files in the hr22 and ki32 project folders on Gadi, the respective folders need to be added to the storage directive in the suite configuration files. +
      + You can do this manually or run the following command from within the suite directory: + 
      grep -rl --exclude-dir=.svn "\-l\s*storage\s*=" . | xargs sed -i '/\-l\s*storage\s*=\s*.*gdata\/hr22.*/! s/\(\-l\s*storage\s*=\s*.*\)/\1+gdata\/hr22/g ; /\-l\s*storage\s*=\s*.*gdata\/ki32.*/! s/\(\-l\s*storage\s*=\s*.*\)/\1+gdata\/ki32/g'
      +
      4. +
    + + +
    + Some suites might not be ported this way. +
    + If you have a suite that was running on accessdev and, even after following the steps above, the run submission fails, consider getting help on the Hive Forum. +
    +
    References
    \ No newline at end of file +