From 4dad2b6baef6725ff11086e9e1720a2ad127223c Mon Sep 17 00:00:00 2001 From: atteggiani Date: Wed, 13 Dec 2023 14:53:07 +1100 Subject: [PATCH 01/17] Updated how to run ACCESS-ESM --- docs/css/access-nri.css | 7 ++ docs/models/run-a-model/run-access-esm.md | 90 +++++++++++++++++------ docs/models/run-a-model/run-access-om.md | 1 + 3 files changed, 77 insertions(+), 21 deletions(-) diff --git a/docs/css/access-nri.css b/docs/css/access-nri.css index 6dfc4da1b..203e819f0 100644 --- a/docs/css/access-nri.css +++ b/docs/css/access-nri.css @@ -1050,6 +1050,13 @@ img.intro-img { margin: 0 0.6em; } +.green_prompt_output { + color: #00C202 +} + +.red_prompt_output { + color: #CA1A01; +} /* =============================================================== Media queries for website responsiveness diff --git a/docs/models/run-a-model/run-access-esm.md b/docs/models/run-a-model/run-access-esm.md index b5bd2f8d5..ef7f7eeee 100644 --- a/docs/models/run-a-model/run-access-esm.md +++ b/docs/models/run-a-model/run-access-esm.md @@ -32,32 +32,80 @@ module load conda/analysis3 ---------------------------------------------------------------------------------------- ## Get {{ model }} configuration -A pre-industrial configuration of {{ model }} is available on the coecms GitHub. +{{ model }} configuration is available on the coecms GitHub.
-To get it on Gadi, create a directory to store the model configuration.Navigate to this directory and clone the GitHub repo in it by running: -
git clone https://github.com/coecms/esm-pre-industrial.git
+To get it on Gadi, clone the {{ model }} GitHub repo by running: +
git clone https://github.com/coecms/access-esm.git
+This will create the access-esm folder. - mkdir -p ~/access-esm - cd ~/access-esm - git clone https://github.com/coecms/esm-pre-industrial - Cloning into 'esm-pre-industrial'... - remote: Enumerating objects: 767, done. - remote: Counting objects: 100% (295/295), done. - remote: Compressing objects: 100% (138/138), done. - remote: Total 767 (delta 173), reused 274 (delta 157), pack-reused 472 - Receiving objects: 100% (767/767), 461.57 KiB | 5.24 MiB/s, done. - Resolving deltas: 100% (450/450), done. + git clone https://github.com/coecms/access-esm.git + Cloning into 'access-esm'... + remote: Enumerating objects: 1625, done. + remote: Counting objects: 100% (1625/1625), done. + remote: Compressing objects: 100% (575/575), done. + remote: Total 1625 (delta 1042), reused 1621 (delta 1040), pack-reused 0 + Receiving objects: 100% (1625/1625), 2.79 MiB | 11.24 MiB/s, done. + Resolving deltas: 100% (1042/1042), done. + ls ~/access-esm + atmosphere config.yaml coupler ice manifests ocean README.md
- Some modules may interfere with git commands (e.g., matlab/R2018a). If you have trouble cloning the repository, run the following command before trying again: 
module purge
+ Some modules may interfere with git commands (e.g., matlab/R2018a). If you have trouble cloning the repository, run the following command before trying again: 
module purge
+ After this step, don't forget to reload the conda/analysis3 module to retrieve payu, as specified in the Model-specific prerequisites section.
+Different {{ model }} configurations are stored in different branches of {{ model }} GitHub repo. +To check all the available branches on the repo, run the following command inside the newly created access-esm folder: +
git branch -a
+ + cd ~/access-esm + git branch -a + ٭ main +   remotes/origin/82ka +   remotes/origin/HEAD → origin/main +   remotes/origin/ccarouge-patch-1 +   remotes/origin/historical +   remotes/origin/last-interglacial +   remotes/origin/last-millenium +   remotes/origin/last-millenium-detailed +   remotes/origin/main +   remotes/origin/mid-holocene +   remotes/origin/pre-industrial +   remotes/origin/ssp585 + +The green-coloured branch (preceded by a star sign `*`) indicates the local branch you are currently in. +
+The red-coloured branches are the available remote branches, formatted as remotes/origin/<branch-name>. +To switch to a specific branch you can run the following command: +
git checkout <branch-name>
+For example, the pre-industrial configuration of {{ model }} is available in the pre-industrial branch. To use the pre-industrial configuration we can run: +
git checkout pre-industrial
+ + git checkout pre-industral + branch 'pre-industrial' set up to track 'origin/pre-industrial'. + Switched to a new branch 'pre-industrial' + git branch +   main + ٭ pre-industrial + ---------------------------------------------------------------------------------------- ## Edit {{ model }} configuration It is good practice to create a new git branch to store all your modifications for a particular run, so as not to modify the reference configuration. -To create a local branch called "example_run", from within the cloned repo execute: -
git checkout -b example_run
+To create a new branch called "example_run", as a copy of the pre-industrial branch, from within the access-esm directory execute: +
git checkout -b example_run $(git log pre-industrial -1 --format=%H)
+This command will also switch to the new example_run branch. + + git branch +   main + ٭ pre-industrial + git checkout -b example_run $(git log pre-industrial -1 --format=%H) + Switched to a new branch 'example_run' + git branch + ٭ example_run +   main +   pre-industrial + ### Payu Payu is a workflow management tool for running numerical models in supercomputing environments. @@ -68,7 +116,7 @@ The general layout of a payu-supported model run consists of two main dir The laboratory directory, where all the model components reside. For {{ model }}, it is typically /scratch/$PROJECT/$USER/access-esm.
  • - The control directory, where the model configuration resides and from where the model is run (in this example, the cloned directory ~/access-esm/esm-pre-industrial). + The control directory, where the model configuration resides and from where the model is run (in this example, the cloned directory ~/access-esm).
    • This distinction of directories separates the small-size configuration files from the larger binary outputs and inputs. In this way, the configuration files can be placed in the $HOME directory (as it is the only filesystem actively backed-up on Gadi), without overloading it with too much data. @@ -83,7 +131,7 @@ This creates the laboratory directory, together with relevant subdirector
  • archive → the directory where output is stored after each run.
    • cd ~/access-esm/esm-pre-industrial - payu init + payu init laboratory path: /scratch/$PROJECT/$USER/access-esm binary path: /scratch/$PROJECT/$USER/access-esm/bin input path: /scratch/$PROJECT/$USER/access-esm/input @@ -102,7 +150,7 @@ This file, which controls the general model configuration, contains several part 
    jobname: pre-industrial
 queue: normal
-walltime: 20:00:00
    +walltime: 3:10:00 These lines can be edited to change the PBS directives for the PBS job.
    For example, to run {{ model }} under the tm70 project (ACCESS-NRI), add the following line: @@ -193,7 +241,7 @@ laboratory: access-esm
  • Number of runs per PBS submission
    - 
    runspersub: 5
    + 
    runspersub: 1
    {{ model }} configurations are often run in multiple steps (or cycles), with payu running a maximum of runspersub internal runs for every PBS job submission.
    If you increase runspersub, you may need to increase the walltime in the PBS resources. @@ -356,7 +404,7 @@ For a complete documentation on how to use this framework, check the Some modules may interfere with git commands (e.g., matlab/R2018a). If you have trouble cloning the repository, run the following command before trying again: 
    module purge
    + After this step, don't forget to reload the conda/analysis3 module to retrieve payu, as specified in the     Model-specific prerequisites section.
    ---------------------------------------------------------------------------------------- From 276d7077c6f86f1f6cfa09917140f47318906154 Mon Sep 17 00:00:00 2001 From: atteggiani Date: Fri, 12 Jan 2024 10:56:49 +1100 Subject: [PATCH 02/17] Updated how-to-run ACCESS-ESM --- docs/models/run-a-model/run-access-esm.md | 12 ++++++++---- 1 file changed, 8 insertions(+), 4 deletions(-) diff --git a/docs/models/run-a-model/run-access-esm.md b/docs/models/run-a-model/run-access-esm.md index ef7f7eeee..ac60ae0f3 100644 --- a/docs/models/run-a-model/run-access-esm.md +++ b/docs/models/run-a-model/run-access-esm.md @@ -80,7 +80,7 @@ To switch to a specific branch you can run the following command: For example, the pre-industrial configuration of {{ model }} is available in the pre-industrial branch. To use the pre-industrial configuration we can run: 
    git checkout pre-industrial
    - git checkout pre-industral + git checkout pre-industrial branch 'pre-industrial' set up to track 'origin/pre-industrial'. Switched to a new branch 'pre-industrial' git branch @@ -404,12 +404,16 @@ For a complete documentation on how to use this framework, check the + A run with a different {{ model }} configuration (different git branch) counts as a new internal run. +
    + Therefore, if output folders were already present, the new output folder will have its internal number set to the first available XXX number. + Outputs and restarts are separated in the respective folders for each model component.
    For the atmospheric output data, the files are usually     UM fieldsfile, formatted as <UM-suite-identifier>a.p<output-stream-identifier><time-identifier>. From 1f3b2f4e0f84d2fad195de387d86e72faf7bcd8a Mon Sep 17 00:00:00 2001 From: atteggiani Date: Fri, 12 Jan 2024 11:10:34 +1100 Subject: [PATCH 03/17] updated setup-python action in workflows from v4 to v5 --- .github/workflows/deploy_to_github_pages.yml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/.github/workflows/deploy_to_github_pages.yml b/.github/workflows/deploy_to_github_pages.yml index 149ee14ce..2dae51bf1 100644 --- a/.github/workflows/deploy_to_github_pages.yml +++ b/.github/workflows/deploy_to_github_pages.yml @@ -73,7 +73,7 @@ jobs: ref: main - name: Python setup - uses: actions/setup-python@v4 + uses: actions/setup-python@v5 with: python-version: 3.x From 55bbd51c292f71b7060500e3c6cba1eed96dac51 Mon Sep 17 00:00:00 2001 From: atteggiani Date: Fri, 12 Jan 2024 11:27:04 +1100 Subject: [PATCH 04/17] updated requirements.txt package versions --- requirements.txt | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/requirements.txt b/requirements.txt index 0b5ae3df8..02049b319 100644 --- a/requirements.txt +++ b/requirements.txt @@ -1,6 +1,6 @@ -mkdocs-material==9.1.8 +mkdocs-material==9.1.21 mkdocs-git-revision-date-localized-plugin==1.2.0 -mkdocs-macros-plugin==0.7.0 -pandoc==2.3 -mkdocs-bibtex==2.8.16 +mkdocs-macros-plugin==1.0.4 +pandoc==3.1.3 +mkdocs-bibtex==2.11.0 git+https://github.com/rbeucher/mkdocs_events_plugin.git \ No newline at end of file From c3199dabd5056fb0a8bf43cedef7edecbb807479 Mon Sep 17 00:00:00 2001 From: atteggiani Date: Fri, 12 Jan 2024 12:00:55 +1100 Subject: [PATCH 05/17] merged development --- .github/workflows/deploy_to_github_pages.yml | 11 ++++++----- docs/models/run-a-model/run-access-cm.md | 4 ++-- requirements.txt | 2 +- 3 files changed, 9 insertions(+), 8 deletions(-) diff --git a/.github/workflows/deploy_to_github_pages.yml b/.github/workflows/deploy_to_github_pages.yml index 2dae51bf1..34a147991 100644 --- a/.github/workflows/deploy_to_github_pages.yml +++ b/.github/workflows/deploy_to_github_pages.yml @@ -69,13 +69,13 @@ jobs: steps: - name: Checkout uses: actions/checkout@v2 - with: - ref: main + # with: + # ref: main - name: Python setup uses: actions/setup-python@v5 with: - python-version: 3.x + python-version: 3.9.x - name: Install dependencies run: pip install -r requirements.txt @@ -106,9 +106,10 @@ jobs: git fetch --all echo "Build main website" + retry "git checkout main" 5 1 "Failed to checkout 'main'." retry 'mkdocs build -f mkdocs.yml -d ../website' 5 1 "Failed to build main website." echo "Build development website" - retry 'git checkout development' 5 1 "Failed to checkout development branch." + retry 'git checkout development' 5 1 "Failed to checkout 'development' branch." retry 'mkdocs build -f mkdocs.yml -d ../website/development-website' 5 1 "Failed to build development website." echo "Build PR websites" command="pr_list=\$(curl -s https://api.github.com/repos/ACCESS-Hive/access-hive.github.io/pulls?state=opened \ @@ -120,7 +121,7 @@ jobs: pr_sha=($(jq '.head.sha' <<< $pr_list)) echo "pr_nums=$(sed 's/\s/,/g' <<< [${pr_nums[@]}])" >> "$GITHUB_OUTPUT" for i in ${!pr_nums[@]}; do - retry "git checkout ${pr_sha[i]}" 5 1 "Failed to checkout git hash ${pr_sha[i]}." + retry "git checkout ${pr_sha[i]}" 5 1 "Failed to checkout git hash '${pr_sha[i]}'." retry "mkdocs build -f mkdocs.yml -d ../website/pr-preview/pr-${pr_nums[i]}" 5 1 "Failed to build pr-${pr_nums[i]} website." done else diff --git a/docs/models/run-a-model/run-access-cm.md b/docs/models/run-a-model/run-access-cm.md index 49e116cb8..831282476 100644 --- a/docs/models/run-a-model/run-access-cm.md +++ b/docs/models/run-a-model/run-access-cm.md @@ -29,7 +29,7 @@ ## Prerequisites ### General prerequisites -Before running {{ model }}, you need to fulfil general prerequisites outlined in the [First Steps](/getting_started/first_steps) section. +Before running {{ model }}, you need to fulfil general prerequisites outlined in the First Steps section. ### Model-specific prerequisites
      @@ -609,7 +609,7 @@ To investigate the cause of a failure, we need to look at the logs job.err ### Model Live Diagnostics -ACCESS-NRI developed the [Model Live Diagnostics](/model_evaluation/model_diagnostics) framework to check, monitor, visualise, and evaluate model behaviour and progress of ACCESS models currently running on Gadi. +ACCESS-NRI developed the Model Live Diagnostics framework to check, monitor, visualise, and evaluate model behaviour and progress of ACCESS models currently running on Gadi.
      For a complete documentation on how to use this framework, check the Model Diagnostics documentation. diff --git a/requirements.txt b/requirements.txt index 02049b319..880cf0ca6 100644 --- a/requirements.txt +++ b/requirements.txt @@ -1,6 +1,6 @@ mkdocs-material==9.1.21 mkdocs-git-revision-date-localized-plugin==1.2.0 mkdocs-macros-plugin==1.0.4 -pandoc==3.1.3 +pandoc==2.3 mkdocs-bibtex==2.11.0 git+https://github.com/rbeucher/mkdocs_events_plugin.git \ No newline at end of file From 231b722171c75514f69ea8b73c2685e29e204727 Mon Sep 17 00:00:00 2001 From: Davide Marchegiani Date: Fri, 12 Jan 2024 14:43:16 +1100 Subject: [PATCH 06/17] Update run-access-esm.md --- docs/models/run-a-model/run-access-esm.md | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/docs/models/run-a-model/run-access-esm.md b/docs/models/run-a-model/run-access-esm.md index ac60ae0f3..1fa9de2cd 100644 --- a/docs/models/run-a-model/run-access-esm.md +++ b/docs/models/run-a-model/run-access-esm.md @@ -53,8 +53,8 @@ This will create the access-esm folder. Some modules may interfere with git commands (e.g., matlab/R2018a). If you have trouble cloning the repository, run the following command before trying again: 
      module purge
      After this step, don't forget to reload the conda/analysis3 module to retrieve payu, as specified in the Model-specific prerequisites section. -Different {{ model }} configurations are stored in different branches of {{ model }} GitHub repo. -To check all the available branches on the repo, run the following command inside the newly created access-esm folder: +Different {{ model }} configurations are stored in different branches of the {{ model }} GitHub repo. +To check all the available branches on the repo, run the following command inside the newly-created access-esm folder: 
      git branch -a
      cd ~/access-esm @@ -77,7 +77,7 @@ The green-coloured branch (preceded by a star sign `*`) indicates the local bran The red-coloured branches are the available remote branches, formatted as remotes/origin/<branch-name>. To switch to a specific branch you can run the following command: 
      git checkout <branch-name>
      -For example, the pre-industrial configuration of {{ model }} is available in the pre-industrial branch. To use the pre-industrial configuration we can run: +For example, the pre-industrial configuration of {{ model }} is available in the pre-industrial branch. To use the pre-industrial configuration, run: 
      git checkout pre-industrial
      git checkout pre-industrial @@ -90,7 +90,7 @@ For example, the pre-industrial configuration of {{ model }} is available in the ---------------------------------------------------------------------------------------- ## Edit {{ model }} configuration -It is good practice to create a new git branch to store all your modifications for a particular run, so as not to modify the reference configuration. +It is good practice to create a new git branch to store all your modifications for a particular run, so as not to modify the reference configuration. To create a new branch called "example_run", as a copy of the pre-industrial branch, from within the access-esm directory execute: 
      git checkout -b example_run $(git log pre-industrial -1 --format=%H)
      @@ -434,4 +434,4 @@ For the atmospheric output data, the files are usually https://payu.readthedocs.io/en/latest/usage.html -
    \ No newline at end of file + From 803c7758f8d9089cc11f3a1856b2e96a5690f5d7 Mon Sep 17 00:00:00 2001 From: Davide Marchegiani Date: Fri, 12 Jan 2024 14:48:30 +1100 Subject: [PATCH 07/17] Update run-access-esm.md --- docs/models/run-a-model/run-access-esm.md | 6 ++---- 1 file changed, 2 insertions(+), 4 deletions(-) diff --git a/docs/models/run-a-model/run-access-esm.md b/docs/models/run-a-model/run-access-esm.md index 1fa9de2cd..8b8178c99 100644 --- a/docs/models/run-a-model/run-access-esm.md +++ b/docs/models/run-a-model/run-access-esm.md @@ -404,15 +404,13 @@ For a complete documentation on how to use this framework, check the A run with a different {{ model }} configuration (different git branch) counts as a new internal run.
    - Therefore, if output folders were already present, the new output folder will have its internal number set to the first available XXX number. + Thus, if output folders already exist, the internal number of the new output folder will be set to the first available XXX number. Outputs and restarts are separated in the respective folders for each model component.
    From 479623298f21549fabe6213465f2172437996d46 Mon Sep 17 00:00:00 2001 From: atteggiani Date: Fri, 19 Jan 2024 14:57:41 +1100 Subject: [PATCH 08/17] First update after claire's comments --- docs/getting_started/first_steps.md | 81 +++++++----- docs/models/run-a-model/run-access-esm.md | 143 ++++++++++++++-------- 2 files changed, 142 insertions(+), 82 deletions(-) diff --git a/docs/getting_started/first_steps.md b/docs/getting_started/first_steps.md index ffd3c4dd9..865d5d576 100644 --- a/docs/getting_started/first_steps.md +++ b/docs/getting_started/first_steps.md @@ -10,9 +10,11 @@ The steps in this section are aimed at new users who would like to do any of the
    ## Create an NCI user account + Most of the data and models you will need are available at the     National Computing Infrastructure (NCI) .
    To access these, you need an NCI account. If you do not have one, sign up here. +
    You will need an institutional email address with an organisation that allows access to NCI (e.g., an Australian university, ACCESS-NRI, CSIRO, BoM, CLEX, etc.).
    @@ -28,23 +30,26 @@ Each project has an ID (e.g. `xp65`), which is what the term project actu If you are interested in datasets and data collections, you can browse the NCI Data Catalogue and follow the NCI Data Catalogue User Guide. -To perform computations on Gadi instead, you need to join a project with computing resources, also called Service Units (SU). The project ID will be provided by your supervisor, research project or institution. +To run models on Gadi instead, you need to join a project with computing resources, also called Service Units (SU). The project ID will be provided by your supervisor, research project or institution. To join a project, search for it on NCI website and request membership. +
    The first project you join will become your default one. If you would like to change this, check out how to change your default project on Gadi. -
    + -There are several NCI projects that may be relevant to you, depending on the tasks you want to carry out. +There are several NCI projects that may be relevant to you, depending on the tasks you want to carry out.
    For tasks supported by ACCESS-NRI (e.g., running a supported model configuration, using a supported model evaluation tool, etc.), you will find a list of relevant projects to join in the pages relative to each respective task.
    ## Login to Gadi + Operations involving model runs and data collections take place on the Gadi supercomputer. Before you login to Gadi, you need to possess the following prerequisites: +
    • Internet connection
    • @@ -62,28 +67,32 @@ Before you login to Gadi, you need to possess the following prerequisites
    To login to Gadi using SSH, on your **local machine's terminal** run the following command, replacing <your-NCI-username> with your NCI username (e.g., ab1234): + 
    ssh <your-NCI-username>@gadi.nci.org.au
    + You will be prompted to enter your NCI password, and then you will be connected to Gadi: - ssh <your-NCI-username>@gadi.nci.org.au - <NCI-username>@gadi.nci.org.au's password: - ############################################################################### - #           Welcome to the NCI National Facility!             # - #    This service is for authorised clients only. It is a criminal      # - #    offence to:                                      # - #          - Obtain access to data without permission;                 # - #          - Damage, delete, alter or insert data without permission;  # - #    Use of this system requires acceptance of the Conditions of Use;    # - #    published at http://nci.org.au/users/nci-terms-and-conditions-access; # - ############################################################################### - |      gadi.nci.org.au - 260,760 processor InfiniBand x86_64 cluster     | - =============================================================================== - =============================================================================== - +ssh <your-NCI-username>@gadi.nci.org.au +<NCI-username>@gadi.nci.org.au's password: +############################################################################### +#           Welcome to the NCI National Facility!             # +#    This service is for authorised clients only. It is a criminal      # +#    offence to:                                      # +#          - Obtain access to data without permission;                 # +#          - Damage, delete, alter or insert data without permission;  # +#    Use of this system requires acceptance of the Conditions of Use;    # +#    published at http://nci.org.au/users/nci-terms-and-conditions-access; # +############################################################################### +|      gadi.nci.org.au - 260,760 processor InfiniBand x86_64 cluster     | +=============================================================================== +=============================================================================== + ### Auto login + To simplify the login and avoid being prompted every time to enter your NCI password, follow these steps: +
    1. Create an SSH key @@ -217,40 +226,46 @@ Once you complete all the above steps, you should be able to connect to Gadi< 
      ssh gadi
      ### Change default project on Gadi + It is recommended that you check what your default project on Gadi is set to. The default project should be set to the computational project you will most likely use to run simulations/evaluations and store data.
      You can check which is your default project by logging into Gadi and running: + 
      echo $PROJECT
      + To change your default project on Gadi, you need to manually change the `PROJECT` field in the `~/.config/gadi-login.conf` file.
      Alternatively, you can run the following command on Gadi's terminal: + 
      sed "s/\(PROJECT \).*/\1<new-default-project>/" ~/.config/gadi-login.conf
      + Once this is done, exit from Gadi and log back in.
      For example, if you want to change your default project to `tm70` on Gadi: - echo $PROJECT +echo $PROJECT <old-default-project> sed "s/\(PROJECT \).*/\1tm70/" ~/.config/gadi-login.conf exit logout Connection to gadi.nci.org.au closed. ssh gadi - ############################################################################### - #           Welcome to the NCI National Facility!             # - #    This service is for authorised clients only. It is a criminal      # - #    offence to:                                      # - #          - Obtain access to data without permission;                 # - #          - Damage, delete, alter or insert data without permission;  # - #    Use of this system requires acceptance of the Conditions of Use;    # - #    published at http://nci.org.au/users/nci-terms-and-conditions-access; # - ############################################################################### - |      gadi.nci.org.au - 260,760 processor InfiniBand x86_64 cluster     | - =============================================================================== - =============================================================================== - echo $PROJECT - tm70 +############################################################################### +#           Welcome to the NCI National Facility!             # +#    This service is for authorised clients only. It is a criminal      # +#    offence to:                                      # +#          - Obtain access to data without permission;                 # +#          - Damage, delete, alter or insert data without permission;  # +#    Use of this system requires acceptance of the Conditions of Use;    # +#    published at http://nci.org.au/users/nci-terms-and-conditions-access; # +############################################################################### +|      gadi.nci.org.au - 260,760 processor InfiniBand x86_64 cluster     | +=============================================================================== +=============================================================================== +echo $PROJECT +tm70 +
      References
      diff --git a/docs/models/run-a-model/run-access-esm.md b/docs/models/run-a-model/run-access-esm.md index 8b8178c99..0b355115d 100644 --- a/docs/models/run-a-model/run-access-esm.md +++ b/docs/models/run-a-model/run-access-esm.md @@ -1,10 +1,17 @@ {% set model = "ACCESS-ESM" %} + # Run {{ model }} + ## Prerequisites + ### General prerequisites + Before running {{ model }}, you need to fulfil general prerequisites outlined in the [First Steps](/getting_started/first_steps) section. ### Model-specific prerequisites + +{{ model }} is installed on NCI's supercomputer Gadi and uses payu, a tool for running and managing model experiments. Following these prerequisites ensures you have access to this infrastructure. +
      • Join the access and hh5 projects at NCI @@ -32,23 +39,27 @@ module load conda/analysis3 ---------------------------------------------------------------------------------------- ## Get {{ model }} configuration -{{ model }} configuration is available on the coecms GitHub. + +{{ model }} configurations are available on the coecms GitHub, collated in a single repository.
        To get it on Gadi, clone the {{ model }} GitHub repo by running: + 
        git clone https://github.com/coecms/access-esm.git
        + This will create the access-esm folder. - git clone https://github.com/coecms/access-esm.git - Cloning into 'access-esm'... - remote: Enumerating objects: 1625, done. - remote: Counting objects: 100% (1625/1625), done. - remote: Compressing objects: 100% (575/575), done. - remote: Total 1625 (delta 1042), reused 1621 (delta 1040), pack-reused 0 - Receiving objects: 100% (1625/1625), 2.79 MiB | 11.24 MiB/s, done. - Resolving deltas: 100% (1042/1042), done. - ls ~/access-esm - atmosphere config.yaml coupler ice manifests ocean README.md +git clone https://github.com/coecms/access-esm.git +Cloning into 'access-esm'... +remote: Enumerating objects: 1625, done. +remote: Counting objects: 100% (1625/1625), done. +remote: Compressing objects: 100% (575/575), done. +remote: Total 1625 (delta 1042), reused 1621 (delta 1040), pack-reused 0 +Receiving objects: 100% (1625/1625), 2.79 MiB | 11.24 MiB/s, done. +Resolving deltas: 100% (1042/1042), done. +ls ~/access-esm +atmosphere config.yaml coupler ice manifests ocean README.md +
        Some modules may interfere with git commands (e.g., matlab/R2018a). If you have trouble cloning the repository, run the following command before trying again: 
        module purge
        After this step, don't forget to reload the conda/analysis3 module to retrieve payu, as specified in the Model-specific prerequisites section. @@ -90,27 +101,32 @@ For example, the pre-industrial configuration of {{ model }} is available in the ---------------------------------------------------------------------------------------- ## Edit {{ model }} configuration -It is good practice to create a new git branch to store all your modifications for a particular run, so as not to modify the reference configuration. + +It is good practice to create a new git branch to store all your modifications for a particular run, so as not to modify the reference configuration. To create a new branch called "example_run", as a copy of the pre-industrial branch, from within the access-esm directory execute: -
        git checkout -b example_run $(git log pre-industrial -1 --format=%H)
        + +
        git checkout -b example_run --no-track origin/pre-industrial
        + This command will also switch to the new example_run branch. - git branch -   main - ٭ pre-industrial - git checkout -b example_run $(git log pre-industrial -1 --format=%H) - Switched to a new branch 'example_run' - git branch - ٭ example_run -   main -   pre-industrial +git branch +  main +٭ pre-industrial +git checkout -b example_run --no-track origin/pre-industrial +Switched to a new branch 'example_run' +git branch +٭ example_run +  main +  pre-industrial ### Payu + Payu is a workflow management tool for running numerical models in supercomputing environments.
        The general layout of a payu-supported model run consists of two main directories: +
        • The laboratory directory, where all the model components reside. For {{ model }}, it is typically /scratch/$PROJECT/$USER/access-esm. @@ -141,9 +157,11 @@ This creates the laboratory directory, together with relevant subdirector
        ### Edit the Master Configuration file -The config.yaml file located in the control directory, is the Master Configuration file. + +The config.yaml file located in the control directory, is the Master Configuration file.
        This file, which controls the general model configuration, contains several parts: +
        • PBS resources @@ -156,7 +174,7 @@ walltime: 3:10:00 For example, to run {{ model }} under the tm70 project (ACCESS-NRI), add the following line: 
          project: tm70
          - To run {{ model }}, you need to be a member of a project with allocated Service Units (SU). For more information, check how to join relevant NCI projects. + The project entry should always refer to a project with allocated Service Units (SU), that you are a member of. If not set explicitly, {{ model }} will run using your default project (this default project still needs to have allocated SU). For more information, check how to join relevant NCI projects.
        • @@ -250,47 +268,63 @@ laboratory: access-esm

        To find out more about other configuration settings for the config.yaml file, check out how to configure your experiment with payu. ----------------------------------------------------------------------------------------- + +### Edit a single {{ model }} component configuration + +Each of {{ model }} components contains additional configuration options that are read in when the model component is running. These options are typically useful to modify the physics used in the model or the input data. +They are specified in the subfolder of the control directory, whose name matches the submodel's name as specified in the config.yaml submodel section (e.g., configuration options for the atmosphere submodel are in the ~/access-esm/esm-pre-industrial/atmosphere directory). +To modify these options please refer to the User Guide of each individual model component. + +--- ## Run {{ model }} configuration -After editing the configuration, you are ready to run {{ model }}. + +After editing the configuration, you are ready to run {{ model }}.
        {{ model }} suites run on Gadi through a PBS job submission managed by payu. ### Payu setup (optional) + As a first step, from within the control directory, it is good practice to run: + 
        payu setup
        + This will prepare the model run, based on the experiment configuration. - payu setup - laboratory path: /scratch/$PROJECT/$USER/access-esm - binary path: /scratch/$PROJECT/$USER/access-esm/bin - input path: /scratch/$PROJECT/$USER/access-esm/input - work path: /scratch/$PROJECT/$USER/access-esm/work - archive path: /scratch/$PROJECT/$USER/access-esm/archive - Loading input manifest: manifests/input.yaml - Loading restart manifest: manifests/restart.yaml - Loading exe manifest: manifests/exe.yaml - Setting up atmosphere - Setting up ocean - Setting up ice - Setting up coupler - Checking exe and input manifests - Updating full hashes for 3 files in manifests/exe.yaml - Creating restart manifest - Updating full hashes for 30 files in manifests/restart.yaml - Writing manifests/restart.yaml - Writing manifests/exe.yaml +payu setup +laboratory path: /scratch/$PROJECT/$USER/access-esm +binary path: /scratch/$PROJECT/$USER/access-esm/bin +input path: /scratch/$PROJECT/$USER/access-esm/input +work path: /scratch/$PROJECT/$USER/access-esm/work +archive path: /scratch/$PROJECT/$USER/access-esm/archive +Loading input manifest: manifests/input.yaml +Loading restart manifest: manifests/restart.yaml +Loading exe manifest: manifests/exe.yaml +Setting up atmosphere +Setting up ocean +Setting up ice +Setting up coupler +Checking exe and input manifests +Updating full hashes for 3 files in manifests/exe.yaml +Creating restart manifest +Updating full hashes for 30 files in manifests/restart.yaml +Writing manifests/restart.yaml +Writing manifests/exe.yaml +
        This step can be skipped as it is also included in the run command. However, running it explicitly helps to check for errors and make sure executable and restart directories are accessible.
        ### Run configuration + To run {{ model }} configuration for one internal run length (controlled by runtime in the config.yaml file), execute: + 
        payu run -f
        + This will submit a single job to the queue with a total run length of runtime. If there is no previous run, it will start from the start date indicated in the config.yaml file. Otherwise, it will perform a warm restart from a previously saved restart file.
        +
        The -f option ensures that payu will run even if there is an existing non-empty work directory created from a previous failed run.
        @@ -305,13 +339,18 @@ This will submit a single job to the queue with a total run length of runt ### Run configuration for multiple years + If you want to run {{ model }} configuration for multiple internal run lengths (controlled by runtime in the config.yaml file), use the option -n: + 
        payu run -f -n <number-of-runs>
        -This will run the configuration number-of-runs times with a total run length of runtime * number-of-runs. The number of consecutive PBS jobs submitted to the queue depends on the runspersub value specified in the config.yaml file. + +This will run the configuration number-of-runs times with a total run length of runtime \* number-of-runs. The number of consecutive PBS jobs submitted to the queue depends on the runspersub value specified in the config.yaml file. ### Understand runtime, runspersub, and -n parameters + With the correct use of runtime, runspersub and -n parameters, you can have full control of your run.
        +
        • runtime defines the internal run length. @@ -363,9 +402,11 @@ days: 0 ---------------------------------------------------------------------------------------- ## Monitor {{ model }} runs -Currently, there is no specific tool to monitor {{ model }} runs. + +Currently, there is no specific tool to monitor {{ model }} runs.
          You can execute the following command to show the status of all your submitted PBS jobs: + 
          qstat -u $USER
          qstat -u $USER @@ -386,27 +427,31 @@ If you changed the jobname in the PBS resources of the Model Diagnostics documentation. ----------------------------------------------------------------------------------------- +--- ## {{ model }} outputs + While the configuration is running, output files (and restart files) are moved from the work directory into the archive directory under /scratch/$PROJECT/$USER/access-esm/archive/access-esm, where they are further subdivided for each internal run. They are also symlinked in the control directory to ~/access-esm/archive
          The naming format for a typical output folder is outputXXX and for a restart folder restartXXX, where XXX is the internal run number starting from 000. +
          A run with a different {{ model }} configuration (different git branch) counts as a new internal run.
          From 9e556ac015eacb613777358ba4ea9505eb358ffe Mon Sep 17 00:00:00 2001 From: atteggiani Date: Fri, 19 Jan 2024 15:12:25 +1100 Subject: [PATCH 09/17] Updated some sections of how to run ACCESS-OM --- docs/models/run-a-model/run-access-esm.md | 4 +- docs/models/run-a-model/run-access-om.md | 111 ++++++++++++++-------- 2 files changed, 76 insertions(+), 39 deletions(-) diff --git a/docs/models/run-a-model/run-access-esm.md b/docs/models/run-a-model/run-access-esm.md index 0b355115d..075c02452 100644 --- a/docs/models/run-a-model/run-access-esm.md +++ b/docs/models/run-a-model/run-access-esm.md @@ -219,7 +219,7 @@ laboratory: access-esm ncpus: 0 input: - /g/data/access/payu/access-esm/input/pre-industrial/coupler - {{ model }} is a coupled model deploying multiple submodels (i.e. model components). + {{ model }} is a coupled model deploying multiple submodels (i.e. model components). This section specifies the submodels and configuration options required to execute the model correctly.
          Each submodel contains additional configuration options that are read in when the submodel is running. These options are specified in the subfolder of the control directory, whose name matches the submodel's name (e.g., configuration options for the atmosphere submodel are in the ~/access-esm/esm-pre-industrial/atmosphere directory). @@ -448,7 +448,7 @@ For a complete documentation on how to use this framework, check the
        • Join the hh5, qv56, ua8 and ik11 projects at NCI @@ -37,7 +42,8 @@ A standard {{ model }} configuration is available on the mkdir -p ~/access-om @@ -58,15 +64,19 @@ To get it on Gadi, create a directory to store the model configuration.Na ---------------------------------------------------------------------------------------- ## Edit {{ model }} configuration -It is good practice to create a new git branch to store all your modifications for a particular run, so as not to modify the reference configuration. + +It is good practice to create a new git branch to store all your modifications for a particular run, so as not to modify the reference configuration. To create a local branch called "example_run", from within the cloned repo execute: + 
          git checkout -b example_run
          ### Payu +           Payu is a workflow management tool for running numerical models in supercomputing environments.
          The general layout of a payu-supported model run consists of two main directories: +
          • The laboratory directory, where all the model components reside. For {{ model }}, it is typically /scratch/$PROJECT/$USER/access-om2. @@ -97,9 +107,11 @@ This creates the laboratory directory, together with relevant subdirector
          ### Edit the Master Configuration file -The config.yaml file located in the control directory, is the Master Configuration file. + +The config.yaml file located in the control directory, is the Master Configuration file.
          This file, which controls the general model configuration, contains several parts: +
          • PBS resources @@ -157,10 +169,10 @@ input: /g/data/ik11/inputs/access-om2/input_20201102/common_1deg_jra55

            - {{ model }} is a coupled model deploying multiple submodels (i.e. model components). + {{ model }} is a coupled model deploying multiple submodels (i.e. model components). This section specifies the submodels and configuration options required to execute the model correctly.
            - Each submodel contains additional configuration options that are read in when the submodel is running. These options are specified in the subfolder of the control directory, whose name matches the submodel's name (e.g., configuration options for the atmosphere submodel are in the ~/access-om/1deg_jra55_iaf/atmosphere directory). + Each submodel contains additional configuration options that are read in when the submodel is running. These options are specified in the subfolder of the control directory, whose name matches the submodel's name (e.g., configuration options for the ocean submodel are in the ~/access-om/1deg_jra55_iaf/ocean directory).
          • Collate @@ -243,8 +255,16 @@ input: /g/data/ik11/inputs/access-om2/input_20201102/common_1deg_jra55

            To find out more about other configuration settings for the config.yaml file, check out how to configure your experiment with payu. +### Edit a single {{ model }} component configuration + +Each of {{ model }} components contains additional configuration options that are read in when the model component is running. These options are typically useful to modify the physics used in the model or the input data. +They are specified in the subfolder of the control directory, whose name matches the submodel's name as specified in the config.yaml submodel section (e.g., configuration options for the ocean submodel are in the ~/access-om/1deg_jra55_iaf/ocean directory). +To modify these options please refer to the User Guide of each individual model component. + ### Change run length + To change the internal run length, edit the restart_period field in the &date_manager_nml section of the ~/access-om/1deg_jra55_iaf/accessom2.nml file: + 
            &date_manager_nml
     forcing_start_date = '1958-01-01T00:00:00'
     forcing_end_date = '2019-01-01T00:00:00'
            
@@ -256,46 +276,55 @@ To change the internal run length, edit the restart_period field in
     The internal run length (controlled by restart_period) can be different from the total run length. Also, while restart_period can be reduced, it should not be increased to more than 5 years to avoid errors. For more information about the difference between internal run and total run lengths, or how to run the model for more than 5 years, refer to the section Run configuration for multiple years.
        ----------------------------------------------------------------------------------------- +--- ## Run {{ model }} configuration -After editing the configuration, you are ready to run {{ model }}. + +After editing the configuration, you are ready to run {{ model }}.
        {{ model }} suites run on Gadi through a PBS job submission managed by payu. ### Payu setup (optional) + As a first step, from within the control directory, it is good practice to run: + 
        payu setup
        + This will prepare the model run, based on the experiment configuration. - payu setup - laboratory path: /scratch/$PROJECT/$USER/access-om2 - binary ppath: /scratch/$PROJECT/$USER/access-om2/bin - input path: /scratch/$PROJECT/$USER/access-om2/input - work path: /scratch/$PROJECT/$USER/access-om2/work - archive path: /scratch/$PROJECT/$USER/access-om2/archive - Loading input manifest: manifests/input.yaml - Loading restart manifest: manifests/restart.yaml - Loading exe manifest: manifests/exe.yaml - Setting up atmosphere - Setting up ocean - Setting up ice - Setting up access-om2 - Checking exe and input manifests - Updating full hashes for 3 files in manifests/exe.yaml - Creating restart manifest - Writing manifests/restart.yaml - Writing manifests/exe.yaml +payu setup +laboratory path: /scratch/$PROJECT/$USER/access-om2 +binary ppath: /scratch/$PROJECT/$USER/access-om2/bin +input path: /scratch/$PROJECT/$USER/access-om2/input +work path: /scratch/$PROJECT/$USER/access-om2/work +archive path: /scratch/$PROJECT/$USER/access-om2/archive +Loading input manifest: manifests/input.yaml +Loading restart manifest: manifests/restart.yaml +Loading exe manifest: manifests/exe.yaml +Setting up atmosphere +Setting up ocean +Setting up ice +Setting up access-om2 +Checking exe and input manifests +Updating full hashes for 3 files in manifests/exe.yaml +Creating restart manifest +Writing manifests/restart.yaml +Writing manifests/exe.yaml +
        This step can be skipped as it is also included in the run command. However, running it explicitly helps to check for errors and make sure executable and restart directories are accessible.
        ### Run configuration + To run {{ model }} configuration for one internal run length (controlled by restart_period in the ~/access-om/1deg_jra55_iaf/accessom2.nml file), execute: + 
        payu run -f
        + This will submit a single job to the queue with a total run length of restart_period.
        +
        The -f option ensures that payu will run even if there is an existing non-empty work directory created from a previous failed run.
        @@ -312,18 +341,23 @@ This will submit a single job to the queue with a total run length of rest ### Run configuration for multiple years + If you want to run {{ model }} configuration for multiple internal run lengths (controlled by restart_period in the ~/access-om/1deg_jra55_iaf/accessom2.nml file), use the option -n: + 
        payu run -f -n <number-of-runs>
        -This will run the configuration number-of-runs times with a total run length of restart_period * number-of-runs. + +This will run the configuration number-of-runs times with a total run length of restart_period \* number-of-runs.
        For example, to run the configuration for a total of 50 years with restart_period = 5, 0, 0 (5 years), the number-of-runs should be set to 10: -
        payu run -f -n 10
        ----------------------------------------------------------------------------------------- + +## 
        payu run -f -n 10
        ## Monitor {{ model }} runs -Currently, there is no specific tool to monitor {{ model }} runs. + +Currently, there is no specific tool to monitor {{ model }} runs.
        You can execute the following command to show the status of all your submitted PBS jobs: + 
        qstat -u $USER
        qstat -u $USER @@ -344,25 +378,28 @@ If you changed the jobname in the PBS resources of the Model Diagnostics documentation. ----------------------------------------------------------------------------------------- +--- ## {{ model }} outputs -While the configuration is running, output files (and restart files) are moved from the work directory to the archive directory /scratch/$PROJECT/$USER/access-om2/archive. They are also symlinked in the control directory to ~/access-om/1deg_jra55_iaf/archive. + +At the end of the model run, output files (and restart files) are moved from the work directory to the archive directory /scratch/$PROJECT/$USER/access-om2/archive. They are also symlinked in the control directory to ~/access-om/1deg_jra55_iaf/archive.
        Both outputs and restarts are stored in subfolders for each different configuration (in this case, 1deg_jra55_iaf). Inside the configuration folder, they are further subdivided for each internal run.
        @@ -371,9 +408,9 @@ The naming format for a typical output folder is outputXXX and for Outputs and restarts are separated in the respective folders for each model component.
        - cd /scratch/$PROJECT/$USER/access-om2/archive/1deg_jra55_iaf - ls - output000 pbs_logs restart000 +cd /scratch/$PROJECT/$USER/access-om2/archive/1deg_jra55_iaf +ls +output000 pbs_logs restart000
        @@ -394,4 +431,4 @@ Outputs and restarts are separated in the respective folders for each model comp
      • https://payu.readthedocs.io/en/latest/usage.html
        • -
      \ No newline at end of file + From bf2257b0e26e0b2cc25bee9bf11673b95d1c3743 Mon Sep 17 00:00:00 2001 From: atteggiani Date: Mon, 22 Jan 2024 09:22:14 +1100 Subject: [PATCH 10/17] Small fix --- docs/models/run-a-model/run-access-om.md | 3 --- 1 file changed, 3 deletions(-) diff --git a/docs/models/run-a-model/run-access-om.md b/docs/models/run-a-model/run-access-om.md index 10dab7091..69a0aa577 100644 --- a/docs/models/run-a-model/run-access-om.md +++ b/docs/models/run-a-model/run-access-om.md @@ -8,11 +8,8 @@ Before running {{ model }}, you need to fulfil general prerequisites outlined in the [First Steps](/getting_started/first_steps) section. -<<<<<<< HEAD -======= If you are unsure whether {{ model }} is the right choice for your experiment, take a look at the overview of [ACCESS Models](/models). ->>>>>>> 756883d74b2c392c02f7c0c5ac6f07acd28fb9b8 ### Model-specific prerequisites
        From d38c99beb78882d5a391d0de7244448eb58410fa Mon Sep 17 00:00:00 2001 From: atteggiani Date: Mon, 22 Jan 2024 12:15:16 +1100 Subject: [PATCH 11/17] test after move --- .github/workflows/delete_all_artifacts.sh | 4 ++-- docs/index.md | 5 +++-- 2 files changed, 5 insertions(+), 4 deletions(-) diff --git a/.github/workflows/delete_all_artifacts.sh b/.github/workflows/delete_all_artifacts.sh index b2411dac7..8599d7661 100755 --- a/.github/workflows/delete_all_artifacts.sh +++ b/.github/workflows/delete_all_artifacts.sh @@ -10,8 +10,8 @@ invoke_api() { } list_artifacts() { - org=ACCESS-Hive - repo=access-hive.github.io + org=ACCESS-NRI + repo=access-hive.org.au invoke_api https://api.github.com/repos/${org}/${repo}/actions/artifacts } diff --git a/docs/index.md b/docs/index.md index eaee2e427..6f9e626a1 100644 --- a/docs/index.md +++ b/docs/index.md @@ -3,10 +3,11 @@ hide: - navigation - toc --- +

        -
        Welcome to ACCESS-Hive!
        +
        Welcome to ACCESS-Hive moved!
        The open portal for the ACCESS user community, hosted by ACCESS-NRI
        @@ -74,4 +75,4 @@ hide: We at ACCESS-NRI acknowledge the Traditional Owners of the land on which our research infrastructure and community operate across Australia and pay our respects to Elders past and present. We recognise the thousands of years of accumulated knowledge and deep connection they have with all the Earth systems we simulate. - \ No newline at end of file + From b3906c4baaefcedf9cf6148c3fc8820ce42d200e Mon Sep 17 00:00:00 2001 From: atteggiani Date: Mon, 22 Jan 2024 12:25:27 +1100 Subject: [PATCH 12/17] Added write permission on PR to allow commenting PRs with the preview link --- .github/workflows/deploy_to_github_pages.yml | 2 ++ docs/index.md | 2 +- 2 files changed, 3 insertions(+), 1 deletion(-) diff --git a/.github/workflows/deploy_to_github_pages.yml b/.github/workflows/deploy_to_github_pages.yml index 816d83772..155ec5f2b 100644 --- a/.github/workflows/deploy_to_github_pages.yml +++ b/.github/workflows/deploy_to_github_pages.yml @@ -12,6 +12,8 @@ on: - closed env: TZ: Australia/Canberra +permissions: + pull-requests: write # to comment on PRs jobs: pr-preview-setup: diff --git a/docs/index.md b/docs/index.md index 6f9e626a1..bcc045a1b 100644 --- a/docs/index.md +++ b/docs/index.md @@ -7,7 +7,7 @@ hide:

        -
        Welcome to ACCESS-Hive moved!
        +
        Welcome to ACCESS-Hive!
        The open portal for the ACCESS user community, hosted by ACCESS-NRI
        From fff3399d0a4d6408c2c14358a8fa39dd26f264d4 Mon Sep 17 00:00:00 2001 From: atteggiani Date: Mon, 22 Jan 2024 15:01:08 +1100 Subject: [PATCH 13/17] Deleted accessdev references --- docs/models/run-a-model/run-access-cm.md | 814 +++++++---------------- 1 file changed, 240 insertions(+), 574 deletions(-) diff --git a/docs/models/run-a-model/run-access-cm.md b/docs/models/run-a-model/run-access-cm.md index 696572c4f..68b257e8f 100644 --- a/docs/models/run-a-model/run-access-cm.md +++ b/docs/models/run-a-model/run-access-cm.md @@ -1,33 +1,6 @@ {% set model = "ACCESS-CM" %} - -
        - - -
        - -
        - -
        -

        Run {{ model }} from ARE / Gadi

        -
        - -
        -

        Run {{ model }} from accessdev

        -
        -
        - - -
        - The workflow to run {{ model }} is currently in transition from accessdev to ARE/Gadi. -
        - The above tabs allow you choose the type of workflow you would prefer to follow. -
        -
        - If you are new to ACCESS-CM, we strongly recommend that you follow the ARE/Gadi workflow, as the accessdev workflow will soon be discontinued. -
        - +# Run {{ model }} ## Prerequisites - ### General prerequisites Before running {{ model }}, you need to fulfil general prerequisites outlined in the First Steps section. @@ -51,262 +24,153 @@ If you are unsure whether {{ model }} is the right choice for your experiment, t
        For more information on how to join specific NCI projects, refer to How to connect to a project. - -
        - -
        -
      • - Connection to an ARE VDI Desktop -
        - To run {{ model }}, you need to be able to start an Australian Research Environment (ARE) VDI Desktop session. -
        - If you are not familiar with ARE, check out the Getting Started on ARE section. -
        • -
        - -
        -
      • - Connection to accessdev -
        - To run {{ model }}, you need to be able to connect to accessdev. This is an NCI server providing configuration and run control for {{ model }}. -
        - You also need to ensure there is correct communication between accessdev and Gadi. -
        - To complete these steps, refer to the SSH & SSH Agent section in the Getting Connected to Accessdev guide. -
        • -
        -
        - +
      • + Connection to an ARE VDI Desktop +
        + To run {{ model }}, you need to be able to start an Australian Research Environment (ARE) VDI Desktop session. +
        + If you are not familiar with ARE, check out the Getting Started on ARE section. +
      -------------------------------------------- -## Setup for your chosen workflow -
      - -
      -
      - Your chosen workflow is ARE / Gadi. If you would prefer to run {{ model }} on accessdev, then select the accessdev workflow. -
      -

      Launch ARE VDI Session

      - Go to the ARE VDI page and launch a session with the following directives: -
        -
      • - Walltime (hours) → ≈ 4 per simulated year. -
        - With the current state of the ARE/Gadi workflow, the ARE VDI session needs to remain active and running for the entirety of the {{ model }} simulation. If the ARE VDI session expires before the end of the simulation, the simulation itself will be terminated as well. -
        -
        - This means that walltime needs to be set according to the simulation length. -
        - A good estimate to calculate the walltime needed is 4 hours per simulated year. -
        - ARE VDI session cannot be spun up for more than 48 consecutive hours. This means that {{ model }} simulations that need more than 48 hours to complete, at the current state, need to be broken down into multiple chunks running for up to 48 hours. -
        -
        - In the near future this will not be necessary anymore, as there will be long running servers in place for runnning {{ model }} simulations. -
        -
        • -
      • - Queue → normalbw -
        • -
      • - Compute Size → tiny (1 CPU) -
        - {{ model }} runs on a different Gadi node with respect to the one where the ARE VDI session is launched. -
        - This means that the ARE VDI session only needs to carry out setup steps as well as starting the run itself. All these tasks can be easily done with only 1 CPU. -
        • -
      • - Project → a project of which you are a member. -
        - The project must have allocated Service Units (SU) to run your simulation. Usually, but not always, this corresponds to your $PROJECT. -
        - For more information, refer to how to Join relevant NCI projects. -
        • -
      • - Storage → gdata/access+gdata/hh5+gdata/hr22+gdata/ki32 (minimum) -
        - This is a list of all project data storage (joined by + signs) needed for the {{ model }} simulation. In ARE, storage locations need to be explicitly defined to access data from within a VDI instance. -
        - Every {{ model }} simulation can be unique and input data can originate from various sources. Hence, if your simulation requires data stored in project folders other than access, hh5, hr22 or ki32, you need to add those projects to the storage path. -
        - For example, if your {{ model }} simulation requires data stored in /g/data/tm70 and /scratch/w40, your full storage path will be: gdata/access+gdata/hh5+gdata/hr22+gdata/ki32+gdata/tm70+scratch/w40 -
        • -
      - Launch the ARE session and, once it starts, click on Launch VDI Desktop. - Launch ARE VDI session +## Setup ARE VDI Desktop +### Launch ARE VDI Session +Go to the ARE VDI page and launch a session with the following directives: +
        +
      • + Walltime (hours) → ≈ 4 per simulated year.
        -

        Open the terminal

        - Once the new tab opens, you will see a Desktop with a few folders on the left. + With the current state of the ARE/Gadi workflow, the ARE VDI session needs to remain active and running for the entirety of the {{ model }} simulation. If the ARE VDI session expires before the end of the simulation, the simulation itself will be terminated as well.
        - To open the terminal, click on the black terminal icon at the top of the window. You should now be connected to a Gadi computing node. - Open ARE VDI terminal -
      - -
      -
      - Your chosen workflow is accessdev. If you would prefer to run {{ model }} on ARE / Gadi, then select the ARE / Gadi workflow. -
      - Login to accessdev by runnning: - 
      ssh accessdev
      +
      + This means that walltime needs to be set according to the simulation length. +
      + A good estimate to calculate the walltime needed is 4 hours per simulated year.
      - If you have not yet set up your accessdev connection through `ssh`, please check the Getting Connected to Accessdev guide. + ARE VDI session cannot be spun up for more than 48 consecutive hours. This means that {{ model }} simulations that need more than 48 hours to complete, at the current state, need to be broken down into multiple chunks running for up to 48 hours. +
      +
      + In the near future this will not be necessary anymore, as there will be long running servers in place for runnning {{ model }} simulations.
      -
      -
      - +
      2. +
    2. + Queue → normalbw +
      3. +
    3. + Compute Size → tiny (1 CPU) +
      + {{ model }} runs on a different Gadi node with respect to the one where the ARE VDI session is launched. +
      + This means that the ARE VDI session only needs to carry out setup steps as well as starting the run itself. All these tasks can be easily done with only 1 CPU. +
      4. +
    4. + Project → a project of which you are a member. +
      + The project must have allocated Service Units (SU) to run your simulation. Usually, but not always, this corresponds to your $PROJECT. +
      + For more information, refer to how to Join relevant NCI projects. +
      5. +
    5. + Storage → gdata/access+gdata/hh5+gdata/hr22+gdata/ki32 (minimum) +
      + This is a list of all project data storage (joined by + signs) needed for the {{ model }} simulation. In ARE, storage locations need to be explicitly defined to access data from within a VDI instance. +
      + Every {{ model }} simulation can be unique and input data can originate from various sources. Hence, if your simulation requires data stored in project folders other than access, hh5, hr22 or ki32, you need to add those projects to the storage path. +
      + For example, if your {{ model }} simulation requires data stored in /g/data/tm70 and /scratch/w40, your full storage path will be: gdata/access+gdata/hh5+gdata/hr22+gdata/ki32+gdata/tm70+scratch/w40 +
      6. + +Launch the ARE session and, once it starts, click on Launch VDI Desktop. +Launch ARE VDI session +
      + +### Open the terminal in the VDI Desktop +Once the new tab opens, you will see a Desktop with a few folders on the left. +
      +To open the terminal, click on the black terminal icon at the top of the window. You should now be connected to a Gadi computing node. +Open ARE VDI terminal ## Get {{ model }} suite {{ model }} comprises the model components UM, MOM, CICE, CABLE and OASIS. These components, which have different model parameters, input data and computer-related information, need to be packaged together as a suite in order to run.
      Each {{ model }} suite has a suite-ID in the format u-<suite-name>, where <suite-name> is a unique identifier.
      -
      - -
      - For this example you can use u-cy339, which is a preindustrial experiment suite. -
      - Typically, an existing suite is copied and then edited as needed for a particular run. -
      - -
      - For this example you can use u-br565, which is the CMIP6-release preindustrial experiment suite. -
      - Typically, an existing suite is copied and then edited as needed for a particular run. -
      -
      - +For this example you can use u-cy339, which is a preindustrial experiment suite. +
      +Typically, an existing suite is copied and then edited as needed for a particular run. + ### Copy {{ model }} suite with Rosie Rosie is an SVN repository wrapper with a set of options specific for {{ model }} suites.
      - -
      - -
      - To copy an existing suite on Gadi you need to follow three main steps: -
        -
      1. - Get Cylc7 setup -
        - To get the Cylc7 setup required to run {{ model }}, execute the following commands: - 
        module use /g/data/hr22/modulefiles
+To copy an existing suite on Gadi you need to follow three main steps:
+
          
+    
        1. 
+        Get Cylc7 setup
+        
          
+        To get the Cylc7 setup required to run {{ model }}, execute the following commands:
+        
          module use /g/data/hr22/modulefiles
 module load cylc7
          
-                
-                    module use /g/data/hr22/modulefiles
-                    module load cylc7
-                    Loading cylc7/23.03
-                     Loading requirement: mosrs-setup/1.0.1
-                
-            
          2. 
+        
+            module use /g/data/hr22/modulefiles
+            module load cylc7
+            Loading cylc7/23.03
+             Loading requirement: mosrs-setup/1.0.1
+        
+    
+    
        2. 
+        MOSRS authentication
+        
          
+        To authenticate using your MOSRS credentials, run:
+        
          mosrs-auth
           
+        
+            mosrs-auth
+            INFO: You need to enter your MOSRS credentials here so that GPG can cache your password.
+            Please enter the MOSRS password for <MOSRS-username>:
+            INFO: Checking your credentials using Subversion. Please wait.
+            INFO: Successfully accessed Subversion with your credentials.
+            INFO: Checking your credentials using rosie. Please wait.
+            INFO: Successfully accessed rosie with your credentials.
+        
+    
          3. 
+    
        3. 
+        Copy a suite
+        
          
+        
            
             
          • 
-                MOSRS authentication
+                Local-only copy
                 
            
-                To authenticate using your MOSRS credentials, run:
-                
            mosrs-auth
             
+                To create a local copy of the <suite-ID> from the UKMO repository, run:
+                
            rosie checkout <suite-ID>
            
                 
-                    mosrs-auth
-                    INFO: You need to enter your MOSRS credentials here so that GPG can cache your password.
-                    Please enter the MOSRS password for <MOSRS-username>:
-                    INFO: Checking your credentials using Subversion. Please wait.
-                    INFO: Successfully accessed Subversion with your credentials.
-                    INFO: Checking your credentials using rosie. Please wait.
-                    INFO: Successfully accessed rosie with your credentials.
+                    rosie checkout <suite-ID>
+                    [INFO] create: /home/565/<$USER>/roses
+                    [INFO] <suite-ID>: local copy created at /home/565/<$USER>/roses/<suite-ID>
                 
+                This option is mostly used for testing and examining existing suites.
             
            • 
             
          • 
-                Copy a suite
-                
            
-                
              
-                    
            • 
-                        Local-only copy
-                        
              
-                        To create a local copy of the <suite-ID> from the UKMO repository, run:
-                        
              rosie checkout <suite-ID>
              
-                        
-                            rosie checkout <suite-ID>
-                            [INFO] create: /home/565/<$USER>/roses
-                            [INFO] <suite-ID>: local copy created at /home/565/<$USER>/roses/<suite-ID>
-                        
-                        This option is mostly used for testing and examining existing suites.
-                    
              • 
-                    
            • 
-                        Remote and local copy
-                        
               
-                        Alternatively, to create a new copy of an existing <suite-ID> both locally and remotely in the UKMO repository, run: 
-                        
              rosie copy <suite-ID>
              
-                        
-                            rosie copy <suite-ID>
-                            Copy "<suite-ID>/trunk@<trunk-ID>" to "u-?????"? [y or n (default)] y
-                            [INFO] <new-suite-ID>: created at https://code.metoffice.gov.uk/svn/roses-u/<suite-n/a/m/e/>
-                            [INFO] <new-suite-ID>: copied items from <suite-ID>/trunk@<trunk-ID>
-                            [INFO] <suite-ID>: local copy created at /home/565/<$USER>/roses/<new-suite-ID>
-                        
-                        When a new suite is created in this way, a unique <suite-ID> is generated within the repository and populated with descriptive information about the suite and its initial configuration.
-                    
              • 
-                
            
-            
            • 
-        
        
-        For additional rosie options, run:
-        
        rosie help
        
-        
        
-        Suites are created in the user's home directory on Gadi under ~/roses/<suite-ID>.
-
      - -
      - To copy an existing suite on accessdev you need to follow two main steps: -
        -
      1. - MOSRS authentication -
        - To authenticate using your MOSRS credentials, run: - 
        mosrs-auth
        + Remote and local copy +
        + Alternatively, to create a new copy of an existing <suite-ID> both locally and remotely in the UKMO repository, run: + 
        rosie copy <suite-ID>
        - mosrs-auth - Please enter the MOSRS password for <MOSRS-username>: - Successfully authenticated with MOSRS as <MOSRS-username> + rosie copy <suite-ID> + Copy "<suite-ID>/trunk@<trunk-ID>" to "u-?????"? [y or n (default)] y + [INFO] <new-suite-ID>: created at https://code.metoffice.gov.uk/svn/roses-u/<suite-n/a/m/e/> + [INFO] <new-suite-ID>: copied items from <suite-ID>/trunk@<trunk-ID> + [INFO] <suite-ID>: local copy created at /home/565/<$USER>/roses/<new-suite-ID> + When a new suite is created in this way, a unique <suite-ID> is generated within the repository and populated with descriptive information about the suite and its initial configuration.
        2. -
      2. - Copy a suite -
        -
          -
        • - Local-only copy -
          - To create a local copy of the <suite-ID> from the UKMO repository, run: - 
          rosie checkout <suite-ID>
          - - rosie checkout <suite-ID> - [INFO] create: /home/565/<$USER>/roses - [INFO] <suite-ID>: local copy created at /home/565/<$USER>/roses/<suite-ID> - - This option is mostly used for testing and examining existing suites. -
          • -
        • - Remote and local copy -
          - Alternatively, to create a new copy of an existing <suite-ID> both locally and remotely in the UKMO repository, run: - 
          rosie copy <suite-ID>
          - - rosie copy <suite-ID> - Copy "<suite-ID>/trunk@<trunk-ID>" to "u-?????"? [y or n (default)] y - [INFO] <new-suite-ID>: created at https://code.metoffice.gov.uk/svn/roses-u/<suite-n/a/m/e/> - [INFO] <new-suite-ID>: copied items from <suite-ID>/trunk@<trunk-ID> - [INFO] <suite-ID>: local copy created at /home/565/<$USER>/roses/<new-suite-ID> - - When a new suite is created in this way, a unique <suite-ID> is generated within the repository and populated with descriptive information about the suite and its initial configuration. -
          • -
        -
        3. -
      - For additional rosie options, run: - 
      rosie help
      -
      - Suites are created in the user's home directory on accessdev under ~/roses/<suite-ID>. -
      -
      - + + +
    +For additional rosie options, run: +
    rosie help
    +
    +Suites are created in the user's home directory on Gadi under ~/roses/<suite-ID>. Each suite directory usually contains two subdirectories and three files:
    • app → directory containing the configuration files for various tasks within the suite.
      • @@ -332,46 +196,19 @@ To edit a suite configuration, run the following command from within the suite d
      The & is optional. It allows the terminal prompt to remain active while running the Rose GUI as a separate process in the background.
      -
      - -
      - - cd ~/roses/<suite-ID> - rose edit & - [<N>] <PID> - - Rose GUI - -
      - -
      - - cd ~/roses/<suite-ID> - rose edit & - [<N>] <PID> - - Rose GUI - -
      -
      - - + + cd ~/roses/<suite-ID> + rose edit & + [<N>] <PID> + + Rose GUI + + ### Change NCI project To ensure that your suite is run under the correct NCI project for which you are a member, edit the Compute project field in suite conf → Machine and Runtime Options, and click the Save button Save button.

      For example, to run an {{ model }} suite under the tm70 project (ACCESS-NRI), enter tm70 in the Compute project field: - -
      - -
      - Rose change project -
      - -
      - Rose change project -
      -
      - +Rose change project
      To run {{ model }}, you need to be a member of a project with allocated Service Units (SU). For more information, check how to Join relevant NCI projects.
      @@ -384,19 +221,7 @@ To modify these parameters, navigate to suite conf → Run Initialisation

      For example, to run a suite for a total of 50 years with a 1-year job resubmission, change Total Run length to P50Y and Cycling frequency to P1Y (the maximum Cycling frequency is currently two years): - - -
      - -
      - Rose change run length -
      - -
      - Rose change run length -
      -
      - +Rose change run length ### Change wallclock time The Wallclock time is the time requested by the PBS job to run a single cycle. If this time is insufficient for the suite to complete a cycle, your job will be terminated before completing the run. Hence, if you change the Cycling frequency, you may also need to change the Wallclock time accordingly. While the time required for a suite to complete a cycle depends on several factors, a good estimation is 4 hours per simulated year. @@ -404,7 +229,6 @@ The Wallclock time is the time requested by the ISO 8601 Duration format) and click the Save button Save button. - ---------------------------------------------------------------------------------------- ## Run {{ model }} suite @@ -422,117 +246,59 @@ To run an {{ model }} suite run the following command from within the suite dire 
      rose suite-run
      After the initial tasks are executed, the Cylc GUI will open. You can now view and control the different tasks in the suite as they are run: - -
      - -
      - - cd ~/roses/<suite-ID> - rose suite-run - [INFO] export CYLC_VERSION=7.9.7 - export ROSE_ORIG_HOST=<gadi-cpu>.gadi.nci.org.au - [INFO] export ROSE_SITE=nci - [INFO] export ROSE_VERSION=2019.01.7 - [INFO] create: /home/565/<$USER>/cylc-run/<suite-ID> - [INFO] create: log.<timestamp> - [INFO] symlink: log.<timestamp> <= log - [INFO] create: log/suite - [INFO] create: log/rose-conf - [INFO] symlink: rose-conf/<timestamp>-run.conf <= log/rose-suite-run.conf - [INFO] symlink: rose-conf/<timestamp>-run.version <= log/rose-suite-run.version - [INFO] create: meta - [INFO] install: meta -     source: /home/565/<$USER>/roses/<suite-ID>/meta - [INFO] install: rose-suite.info -     source: /home/565/<$USER>/roses/<suite-ID>/rose-suite.info - [INFO] create: app - [INFO] install: app -     source: /home/565/<$USER>/roses/<suite-ID>/app - [INFO] install: suite.rc - [INFO] REGISTERED <suite-ID> -> /home/565/<$USER>/cylc-run/<suite-ID> - [INFO] create: share - [INFO] create: share/cycle - [INFO] create: work - [INFO] chdir: log/ - [INFO]         ._. - [INFO]         | |         The Cylc Suite Engine [7.9.7] - [INFO] ._____._. ._| |_____.      Copyright (C) 2008-2019 NIWA - [INFO] | .___| | | | | .___| & British Crown (Met Office) & Contributors. - [INFO] | !___| !_! | | !___. _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ - [INFO] !_____!___. |_!_____! This program comes with ABSOLUTELY NO WARRANTY; - [INFO]     .___! |      see `cylc warranty`.  It is free software, you - [INFO]     !_____!       are welcome to redistribute it under certain - [INFO] - [INFO] *** listening on https://<gadi-cpu>.gadi.nci.org.au:<port>/ *** - [INFO] - [INFO] To view suite server program contact information: - [INFO] $ cylc get-suite-contact <suite-ID> - [INFO] - [INFO] Other ways to see if the suite is still running: - [INFO] $ cylc scan -n '<suite-ID>' <gadi-cpu>.gadi.nci.org.au - [INFO] $ cylc ping -v --host=<gadi-cpu>.nci.org.au <suite-ID> - [INFO] $ ps -opid,args <PID> # on <gadi-cpu>.nci.org.au - Cylc GUI - -
      - -
      - - cd ~/roses/<suite-ID> - rose suite-run - [INFO] export CYLC_VERSION=7.8.3 - [INFO] export ROSE_ORIG_HOST=accessdev.nci.org.au - [INFO] export ROSE_SITE= - [INFO] export ROSE_VERSION=2019.01.2 - [INFO] create: /home/565/<$USER>/cylc-run/<suite-ID> - [INFO] create: log.<timestamp> - [INFO] symlink: log.<timestamp> <= log - [INFO] create: log/suite - [INFO] create: log/rose-conf - [INFO] symlink: rose-conf/<timestamp>-run.conf <= log/rose-suite-run.conf - [INFO] symlink: rose-conf/<timestamp>-run.version <= log/rose-suite-run.version - [INFO] install: rose-suite.info -     source: /home/565/<$USER>/roses/<suite-ID>/rose-suite.info - [INFO] create: app - [INFO] install: app -     source: /home/565/<$USER>/roses/<suite-ID>/app - [INFO] create: meta - [INFO] install: meta -     source: /home/565/<$USER>/roses/<suite-ID>/meta - [INFO] install: suite.rc - [INFO] REGISTERED <suite-ID> -> /home/565/<$USER>/cylc-run/<suite-ID> - [INFO] create: share - [INFO] install: share - [INFO] create: work - [INFO] chdir: log/ - [INFO]         ._. - [INFO]         | |         The Cylc Suite Engine [7.8.3] - [INFO] ._____._. ._| |_____.      Copyright (C) 2008-2019 NIWA - [INFO] | .___| | | | | .___| & British Crown (Met Office) & Contributors. - [INFO] | !___| !_! | | !___. _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ - [INFO] !_____!___. |_!_____! This program comes with ABSOLUTELY NO WARRANTY; - [INFO]     .___! |      see `cylc warranty`.  It is free software, you - [INFO]     !_____!       are welcome to redistribute it under certain - [INFO] - [INFO] *** listening on https://accessdev.nci.org.au:<port>/ *** - [INFO] - [INFO] To view suite server program contact information: - [INFO] $ cylc get-suite-contact <suite-ID> - [INFO] - [INFO] Other ways to see if the suite is still running: - [INFO] $ cylc scan -n '<suite-ID>' accessdev.nci.org.au - [INFO] $ cylc ping -v --host=accessdev.nci.org.au <suite-ID> - [INFO] $ ps -opid,args <PID> # on accessdev.nci.org.au - Cylc GUI - -
      -
      - + + cd ~/roses/<suite-ID> + rose suite-run + [INFO] export CYLC_VERSION=7.9.7 + export ROSE_ORIG_HOST=<gadi-cpu>.gadi.nci.org.au + [INFO] export ROSE_SITE=nci + [INFO] export ROSE_VERSION=2019.01.7 + [INFO] create: /home/565/<$USER>/cylc-run/<suite-ID> + [INFO] create: log.<timestamp> + [INFO] symlink: log.<timestamp> <= log + [INFO] create: log/suite + [INFO] create: log/rose-conf + [INFO] symlink: rose-conf/<timestamp>-run.conf <= log/rose-suite-run.conf + [INFO] symlink: rose-conf/<timestamp>-run.version <= log/rose-suite-run.version + [INFO] create: meta + [INFO] install: meta +     source: /home/565/<$USER>/roses/<suite-ID>/meta + [INFO] install: rose-suite.info +     source: /home/565/<$USER>/roses/<suite-ID>/rose-suite.info + [INFO] create: app + [INFO] install: app +     source: /home/565/<$USER>/roses/<suite-ID>/app + [INFO] install: suite.rc + [INFO] REGISTERED <suite-ID> -> /home/565/<$USER>/cylc-run/<suite-ID> + [INFO] create: share + [INFO] create: share/cycle + [INFO] create: work + [INFO] chdir: log/ + [INFO]         ._. + [INFO]         | |         The Cylc Suite Engine [7.9.7] + [INFO] ._____._. ._| |_____.      Copyright (C) 2008-2019 NIWA + [INFO] | .___| | | | | .___| & British Crown (Met Office) & Contributors. + [INFO] | !___| !_! | | !___. _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ + [INFO] !_____!___. |_!_____! This program comes with ABSOLUTELY NO WARRANTY; + [INFO]     .___! |      see `cylc warranty`.  It is free software, you + [INFO]     !_____!       are welcome to redistribute it under certain + [INFO] + [INFO] *** listening on https://<gadi-cpu>.gadi.nci.org.au:<port>/ *** + [INFO] + [INFO] To view suite server program contact information: + [INFO] $ cylc get-suite-contact <suite-ID> + [INFO] + [INFO] Other ways to see if the suite is still running: + [INFO] $ cylc scan -n '<suite-ID>' <gadi-cpu>.gadi.nci.org.au + [INFO] $ cylc ping -v --host=<gadi-cpu>.nci.org.au <suite-ID> + [INFO] $ ps -opid,args <PID> # on <gadi-cpu>.nci.org.au + Cylc GUI +
      After running the command rose suite-run, if you get an error similar to the following: 
      [FAIL] Suite "<suite-ID>" appears to be running:
 [FAIL] Contact info from: "/home/565/<$USER>/cylc-run/<suite-ID>/.service/contact"
-[FAIL]    CYLC_SUITE_HOST=accessdev.nci.org.au
+[FAIL]    CYLC_SUITE_HOST=gadi.nci.org.au
 [FAIL]    CYLC_SUITE_OWNER=<$USER>
 [FAIL]    CYLC_SUITE_PORT=<port>
 [FAIL]    CYLC_SUITE_PROCESS=<PID> python2 /usr/local/cylc/cylc-7.8.3/bin/cylc-run <suite-ID>
@@ -563,17 +329,7 @@ To investigate the cause of a failure, we need to look at the logs job.err
         
      
         To access a specific task, click on the arrow next to the task to extend the drop-down menu with all the subtasks.
         
      
-        
      
-            
-            
      
-                Investigate Error GUI
-            
      
-            
-            
      
-                Investigate Error GUI
-            
      
-        
      
-        
+        Investigate Error GUI
     
     
    • 
         Through the suite directory
@@ -626,29 +382,13 @@ To scan for active suites, run:
 
      cylc scan
      
 To reopen the Cylc GUI, run the following command from within the suite directory:
 
      rose suite-gcontrol
      
-
      
-    
-    
      
-        
-            cylc scan
-            <suite-ID> <$USER>@<gadi-cpu>.nci.org.au:<port>
-            cd ~/roses/<suite-ID>
-            rose suite-gcontrol
-            Cylc GUI
-        
-    
      
-    
-    
      
-        
-            cylc scan
-            <suite-ID> <$USER>@accessdev.nci.org.au:<port>
-            cd ~/roses/<suite-ID>
-            rose suite-gcontrol
-            Cylc GUI
-        
-    
      
-
      
-
+
+    cylc scan
+    <suite-ID> <$USER>@<gadi-cpu>.nci.org.au:<port>
+    cd ~/roses/<suite-ID>
+    rose suite-gcontrol
+    Cylc GUI
+
 
 ### STOP a suite
 To shutdown a suite in a safe manner, run the following command from within the suite directory:
@@ -676,81 +416,39 @@ There are two main ways to restart a suite:
         
      
             You may need to manually trigger failed tasks from the Cylc GUI.
         
      
-        
      
-            
-            
      
-                
-                    cylc
-                    cd ~/roses/<suite-ID>
-                    rose suite-run --restart
-                    [INFO] export CYLC_VERSION=7.9.7
-                    [INFO] export ROSE_ORIG_HOST=<gadi-cpu>.nci.org.au
-                    [INFO] export ROSE_SITE=nci
-                    [INFO] export ROSE_VERSION=2019.01.2
-                    [INFO] delete: log/rose-suite-run.conf
-                    [INFO] symlink: rose-conf/<timestamp>-restart.conf <= log/rose-suite-run.conf
-                    [INFO] delete: log/rose-suite-run.version
-                    [INFO] symlink: rose-conf/<timestamp>-restart.version <= log/rose-suite-run.version
-                    [INFO] chdir: log/
-                    [INFO]         ._.
-                    [INFO]         | |         The Cylc Suite Engine [7.9.7]
-                    [INFO] ._____._. ._| |_____.      Copyright (C) 2008-2019 NIWA
-                    [INFO] | .___| | | | | .___| & British Crown (Met Office) & Contributors.
-                    [INFO] | !___| !_! | | !___. _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
-                    [INFO] !_____!___. |_!_____! This program comes with ABSOLUTELY NO WARRANTY;
-                    [INFO]     .___! |      see `cylc warranty`.  It is free software, you
-                    [INFO]     !_____!       are welcome to redistribute it under certain
-                    [INFO]
-                    [INFO] *** listening on https://<gadi-cpu>.nci.org.au:<port>/ ***
-                    [INFO]
-                    [INFO] To view suite server program contact information:
-                    [INFO]  $ cylc get-suite-contact <suite-ID>
-                    [INFO]
-                    [INFO] Other ways to see if the suite is still running:
-                    [INFO]  $ cylc scan -n '<suite-ID>' <gadi-cpu>.nci.org.au
-                    [INFO]  $ cylc ping -v --host=<gadi-cpu>.nci.org.au <suite-ID>
-                    [INFO]  $ ps -opid,args <PID>  # on <gadi-cpu>.nci.org.au
-                    Cylc GUI
-                
-            
      
-            
-            
      
-                
-                    cylc
-                    cd ~/roses/<suite-ID>
-                    rose suite-run --restart
-                    [INFO] export CYLC_VERSION=7.8.3
-                    [INFO] export ROSE_ORIG_HOST=accessdev.nci.org.au
-                    [INFO] export ROSE_SITE=
-                    [INFO] export ROSE_VERSION=2019.01.2
-                    [INFO] delete: log/rose-suite-run.conf
-                    [INFO] symlink: rose-conf/<timestamp>-restart.conf <= log/rose-suite-run.conf
-                    [INFO] delete: log/rose-suite-run.version
-                    [INFO] symlink: rose-conf/<timestamp>-restart.version <= log/rose-suite-run.version
-                    [INFO] chdir: log/
-                    [INFO]         ._.
-                    [INFO]         | |         The Cylc Suite Engine [7.8.3]
-                    [INFO] ._____._. ._| |_____.      Copyright (C) 2008-2019 NIWA
-                    [INFO] | .___| | | | | .___| & British Crown (Met Office) & Contributors.
-                    [INFO] | !___| !_! | | !___. _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
-                    [INFO] !_____!___. |_!_____! This program comes with ABSOLUTELY NO WARRANTY;
-                    [INFO]     .___! |      see `cylc warranty`.  It is free software, you
-                    [INFO]     !_____!       are welcome to redistribute it under certain
-                    [INFO]
-                    [INFO] *** listening on https://accessdev.nci.org.au:<port>/ ***
-                    [INFO]
-                    [INFO] To view suite server program contact information:
-                    [INFO]  $ cylc get-suite-contact <suite-ID>
-                    [INFO]
-                    [INFO] Other ways to see if the suite is still running:
-                    [INFO]  $ cylc scan -n '<suite-ID>' accessdev.nci.org.au
-                    [INFO]  $ cylc ping -v --host=accessdev.nci.org.au <suite-ID>
-                    [INFO]  $ ps -opid,args <PID>  # on accessdev.nci.org.au
-                    Cylc GUI
-                
-            
      
-        
      
-        
+        
+            cylc
+            cd ~/roses/<suite-ID>
+            rose suite-run --restart
+            [INFO] export CYLC_VERSION=7.9.7
+            [INFO] export ROSE_ORIG_HOST=<gadi-cpu>.nci.org.au
+            [INFO] export ROSE_SITE=nci
+            [INFO] export ROSE_VERSION=2019.01.2
+            [INFO] delete: log/rose-suite-run.conf
+            [INFO] symlink: rose-conf/<timestamp>-restart.conf <= log/rose-suite-run.conf
+            [INFO] delete: log/rose-suite-run.version
+            [INFO] symlink: rose-conf/<timestamp>-restart.version <= log/rose-suite-run.version
+            [INFO] chdir: log/
+            [INFO]         ._.
+            [INFO]         | |         The Cylc Suite Engine [7.9.7]
+            [INFO] ._____._. ._| |_____.      Copyright (C) 2008-2019 NIWA
+            [INFO] | .___| | | | | .___| & British Crown (Met Office) & Contributors.
+            [INFO] | !___| !_! | | !___. _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
+            [INFO] !_____!___. |_!_____! This program comes with ABSOLUTELY NO WARRANTY;
+            [INFO]     .___! |      see `cylc warranty`.  It is free software, you
+            [INFO]     !_____!       are welcome to redistribute it under certain
+            [INFO]
+            [INFO] *** listening on https://<gadi-cpu>.nci.org.au:<port>/ ***
+            [INFO]
+            [INFO] To view suite server program contact information:
+            [INFO]  $ cylc get-suite-contact <suite-ID>
+            [INFO]
+            [INFO] Other ways to see if the suite is still running:
+            [INFO]  $ cylc scan -n '<suite-ID>' <gadi-cpu>.nci.org.au
+            [INFO]  $ cylc ping -v --host=<gadi-cpu>.nci.org.au <suite-ID>
+            [INFO]  $ ps -opid,args <PID>  # on <gadi-cpu>.nci.org.au
+            Cylc GUI
+        
     
      • 
     
        
     
    • 
@@ -802,62 +500,30 @@ This directory contains two subdirectories:
     
      •
    For the atmospheric output data, the files are typically a UM fieldsfile or netCDF file, formatted as <suite-name>a.p<output-stream-identifier><year><month-string>. -
    -
    - -
    - For the u-cy339 suite in this example, the atm directory contains: - - cd /scratch/<$PROJECT>/<$USER>/archive - ls - cy339 <other-suite-name> <other-suite-name> - cd cy339 - ls - history restart - ls history/atm - cy339a.pd0950apr.nc cy339a.pd0950aug.nc cy339a.pd0950dec.nc cy339a.pd0950feb.nc cy339a.pd0950jan.nc cy339a.pd0950jul.nc cy339a.pd0950jun.nc cy339a.pd0950mar.nc cy339a.pd0950may.nc cy339a.pd0950nov.nc cy339a.pd0950oct.nc cy339a.pd0950sep.nc cy339a.pd0951apr.nc cy339a.pd0951aug.nc cy339a.pd0951dec.nc cy339a.pm0950apr.nc cy339a.pm0950aug.nc cy339a.pm0950dec.nc cy339a.pm0950feb.nc cy339a.pm0950jan.nc cy339a.pm0950jul.nc cy339a.pm0950jun.nc cy339a.pm0950mar.nc cy339a.pm0950may.nc cy339a.pm0950nov.nc cy339a.pm0950oct.nc cy339a.pm0950sep.nc cy339a.pm0951apr.nc cy339a.pm0951aug.nc cy339a.pm0951dec.nc netCDF - -
    - -
    - For the u-br565 suite in this example, the atm directory contains: - - cd /scratch/<$PROJECT>/<$USER>/archive - ls - br565 <other-suite-name> <other-suite-name> - cd br565 - ls - history restart - ls history/atm - br565a.pd0950apr.nc br565a.pd0950aug.nc br565a.pd0950dec.nc br565a.pd0950feb.nc br565a.pd0950jan.nc br565a.pd0950jul.nc br565a.pd0950jun.nc br565a.pd0950mar.nc br565a.pd0950may.nc br565a.pd0950nov.nc br565a.pd0950oct.nc br565a.pd0950sep.nc br565a.pd0951apr.nc br565a.pd0951aug.nc br565a.pd0951dec.nc br565a.pm0950apr.nc br565a.pm0950aug.nc br565a.pm0950dec.nc br565a.pm0950feb.nc br565a.pm0950jan.nc br565a.pm0950jul.nc br565a.pm0950jun.nc br565a.pm0950mar.nc br565a.pm0950may.nc br565a.pm0950nov.nc br565a.pm0950oct.nc br565a.pm0950sep.nc br565a.pm0951apr.nc br565a.pm0951aug.nc br565a.pm0951dec.nc netCDF - -
    -
    - + +For the u-cy339 suite in this example, the atm directory contains: + + cd /scratch/<$PROJECT>/<$USER>/archive + ls + cy339 <other-suite-name> <other-suite-name> + cd cy339 + ls + history restart + ls history/atm + cy339a.pd0950apr.nc cy339a.pd0950aug.nc cy339a.pd0950dec.nc cy339a.pd0950feb.nc cy339a.pd0950jan.nc cy339a.pd0950jul.nc cy339a.pd0950jun.nc cy339a.pd0950mar.nc cy339a.pd0950may.nc cy339a.pd0950nov.nc cy339a.pd0950oct.nc cy339a.pd0950sep.nc cy339a.pd0951apr.nc cy339a.pd0951aug.nc cy339a.pd0951dec.nc cy339a.pm0950apr.nc cy339a.pm0950aug.nc cy339a.pm0950dec.nc cy339a.pm0950feb.nc cy339a.pm0950jan.nc cy339a.pm0950jul.nc cy339a.pm0950jun.nc cy339a.pm0950mar.nc cy339a.pm0950may.nc cy339a.pm0950nov.nc cy339a.pm0950oct.nc cy339a.pm0950sep.nc cy339a.pm0951apr.nc cy339a.pm0951aug.nc cy339a.pm0951dec.nc netCDF + ### Restart files The restart files can be found in the /scratch/$PROJECT/$USER/archive/<suite-name>/restart directory, where they are categorised according to model components (similar to the history folder above).
    The atmospheric restart files, which are UM fieldsfiles, are formatted as <suite-name>a.da<year><month><day>_00. -
    - -
    - For the u-cy339 suite in this example, the atm directory contains: - - ls /scratch/<$PROJECT>/<$USER>/archive/cy339/restart/atm - cy339a.da09500201_00 cy339a.da09510101_00 cy339.xhist-09500131 cy339.xhist-09501231 - -
    - -
    - For the u-br565 suite in this example, the atm directory contains: - - ls /scratch/<$PROJECT>/<$USER>/archive/br565/restart/atm - br565a.da09500201_00 br565a.da09510101_00 br565.xhist-09500131 br565.xhist-09501231 - -
    -
    - + +For the u-cy339 suite in this example, the atm directory contains: + + ls /scratch/<$PROJECT>/<$USER>/archive/cy339/restart/atm + cy339a.da09500201_00 cy339a.da09510101_00 cy339.xhist-09500131 cy339.xhist-09501231 + + Files formatted as <suite-name>a.xhist-<year><month><day> contain metadata information.
    From 297d9ba2b8d047e20c39a4014e8d61601d5f92e5 Mon Sep 17 00:00:00 2001 From: atteggiani Date: Tue, 30 Jan 2024 13:50:22 +1100 Subject: [PATCH 14/17] Updated 'How to run ACCESS-CM' page with persistent sessions workflow. --- docs/js/miscellaneous.js | 2 +- docs/models/run-a-model/run-access-cm.md | 214 ++++++++++++++++++----- 2 files changed, 175 insertions(+), 41 deletions(-) diff --git a/docs/js/miscellaneous.js b/docs/js/miscellaneous.js index 46245f300..9f25cfedb 100644 --- a/docs/js/miscellaneous.js +++ b/docs/js/miscellaneous.js @@ -62,7 +62,7 @@ function tabFunctionality() { document.querySelectorAll('[href^="#"]:not([class^="md"])').forEach(el => { let href = el.getAttribute('href'); let tabEl = document.getElementById(href.slice(1,)) - if (tabEl.parentElement.classList.contains("tabLabels")) { + if (tabEl?.parentElement.classList.contains("tabLabels")) { el.addEventListener("click",(e) => openTab(tabEl), false); } }) diff --git a/docs/models/run-a-model/run-access-cm.md b/docs/models/run-a-model/run-access-cm.md index 68b257e8f..fb2f69e31 100644 --- a/docs/models/run-a-model/run-access-cm.md +++ b/docs/models/run-a-model/run-access-cm.md @@ -1,5 +1,10 @@ {% set model = "ACCESS-CM" %} # Run {{ model }} +
    + Important for accessdev users! +
    + If you are an accessdev user, make sure you are a member of hr22 and ki32 projects, and then check the new persistent session worflow for {{ model }}, and how to port suites from accessdev. +
    ## Prerequisites ### General prerequisites Before running {{ model }}, you need to fulfil general prerequisites outlined in the First Steps section. @@ -25,34 +30,29 @@ If you are unsure whether {{ model }} is the right choice for your experiment, t For more information on how to join specific NCI projects, refer to How to connect to a project.
  • - Connection to an ARE VDI Desktop + Connection to an ARE VDI Desktop (optional)
    - To run {{ model }}, you need to be able to start an Australian Research Environment (ARE) VDI Desktop session. + To run {{ model }}, it is preferred to start an Australian Research Environment (ARE) VDI Desktop session.
    If you are not familiar with ARE, check out the Getting Started on ARE section.
    • -------------------------------------------- -## Setup ARE VDI Desktop +## Setup ARE VDI Desktop (optional) +To skip this step and run {{ model }} from Gadi login node instead, check Setup {{ model }} persistent session. + ### Launch ARE VDI Session Go to the ARE VDI page and launch a session with the following directives:
    • - Walltime (hours) → ≈ 4 per simulated year. + Walltime (hours) → 2
      - With the current state of the ARE/Gadi workflow, the ARE VDI session needs to remain active and running for the entirety of the {{ model }} simulation. If the ARE VDI session expires before the end of the simulation, the simulation itself will be terminated as well. + This is the amount of time the ARE VDI session will stay active for.
      + {{ model }} does not run directly on ARE.
      - This means that walltime needs to be set according to the simulation length. -
      - A good estimate to calculate the walltime needed is 4 hours per simulated year. -
      - ARE VDI session cannot be spun up for more than 48 consecutive hours. This means that {{ model }} simulations that need more than 48 hours to complete, at the current state, need to be broken down into multiple chunks running for up to 48 hours. -
      -
      - In the near future this will not be necessary anymore, as there will be long running servers in place for runnning {{ model }} simulations. -
      + This means that the ARE VDI session only needs to carry out setup steps as well as starting the run itself. All these tasks can be done within 2 hours.
    • Queue → normalbw @@ -60,9 +60,7 @@ Go to the +## Setup {{ model }} persistent session +To support the use of long-running processes (such as ACCESS models runs), NCI provides a service on Gadi called persistent sessions. + +To run {{ model }}, you need to start a persistent session and set it as the target session for the model run. + +### Start a new persistent session +To start a new persistent session on Gadi (in a login node, or in an ARE terminal instance), run the following command: +
      persistent-sessions start <name>
      + +This will start a persistent session with the given name that runs under your default project. +
      +If you want to assign a different project to the persistent session, use the option -p: +
      persistent-sessions start -p <project> <name>
      + +
      + The project assigned to a persistent session does not have to be necessarily the same to the project used to run {{ model }} configuration, but needs to have allocated Service Units (SU). +
      + For more information, check how to Join relevant NCI projects. +
      + + persistent-sessions start <name> + session <persistent-session-uuid> running - connect using +  ssh <name>.<$USER>.<project>.ps.gadi.nci.org.au + + +To list all the active persistent sessions run: +
      persistent-sessions list
      + + + persistent-sessions list +              UUID  PROJECT   ADDRESS    CPUTIME MEMORY + <persistent-session-uuid> <project> 10.9.0.62 00:00:05.213 30.5M + + + +The full name of a newly created persistent session is formatted as: +
      +<name>.<$USER>.<project>.ps.gadi.nci.org.au. + +### Specify {{ model }} target persistent session + +After starting the persistent session, it is essential to assign it to the {{ model }} run. +
      +The easiest way to do so, is to insert the persistent session full name into the file ~/.persistent-sessions/cylc-session. +
      +You can do it manually, or by running the following command: + +
      cat > ~/.persistent-sessions/cylc-session <<< <name>.<$USER>.<project>.ps.gadi.nci.org.au
      + +For example, if the user dm5220 started a persistent session named cylc, under the project tm70, the command will be: + + + cat > ~/.persistent-sessions/cylc-session <<< cylc.dm5220.tm70.ps.gadi.nci.org.au + cat ~/.persistent-sessions/cylc-session + cylc.dm5220.tm70.ps.gadi.nci.org.au + + +For more information on how to specify the target session, check Specify Target Session on Cylc7 Suites. +
      + You can concurrently submit multiple {{ model }} runs using the same persistent session, without the need to start another one. Therefore, the process of specifying the target persistent session for {{ model }} can potentially be done only once. +
      + After specifying the {{ model }} target persistent session the first time, to run {{ model }} you just have to make sure to have an active persistent session named like the {{ model }} target persistent session. +
      + +### Terminate a persistent session +To stop a persistent session, run: +
      persistent-sessions kill <persistent-session-uuid>
      +
      + When you terminate a persistent session, any model running on that session will stop. Therefore, you should check whether you have any active model runs before terminating a persistent session. +
      + + ## Get {{ model }} suite {{ model }} comprises the model components UM, MOM, CICE, CABLE and OASIS. These components, which have different model parameters, input data and computer-related information, need to be packaged together as a suite in order to run.
      @@ -100,7 +170,6 @@ For this example you can use u-cy339, which is a preindustrial expe
      Typically, an existing suite is copied and then edited as needed for a particular run. - ### Copy {{ model }} suite with Rosie Rosie is an SVN repository wrapper with a set of options specific for {{ model }} suites.
      @@ -111,14 +180,21 @@ To copy an existing suite on Gadi you need to follow three main steps:
      To get the Cylc7 setup required to run {{ model }}, execute the following commands: 
      module use /g/data/hr22/modulefiles
-module load cylc7
      +module load cylc7/23.09       module use /g/data/hr22/modulefiles - module load cylc7 - Loading cylc7/23.03 + module load cylc7/23.09 + Using the cylc session <name>.<$USER>.<project>.ps.gadi.nci.org.au + + Loading cylc7/23.09  Loading requirement: mosrs-setup/1.0.1
      • +
      + Make sure to load a version of Cylc >= 23.09, as earlier versions do not support the persistent sessions workflow. +
      + Also, before loading the Cylc module, make sure to have started a persistent session and to have assigned it to the {{ model }} workflow. For more information about these steps, check Setup {{ model }} persistent session. +
    • MOSRS authentication
      @@ -274,34 +350,39 @@ After the initial tasks are executed, the Cylc GUI will open. You can now [INFO] create: share/cycle [INFO] create: work [INFO] chdir: log/ - [INFO]         ._. + [WARN] Using the cylc session <persistent-session-full-name> + [WARN] + [WARN] Loading cylc7/23.09 + [WARN]  Loading requirement: mosrs-setup/1.0.1 + [INFO]         ._. [INFO]         | |         The Cylc Suite Engine [7.9.7] - [INFO] ._____._. ._| |_____.      Copyright (C) 2008-2019 NIWA - [INFO] | .___| | | | | .___| & British Crown (Met Office) & Contributors. - [INFO] | !___| !_! | | !___. _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ - [INFO] !_____!___. |_!_____! This program comes with ABSOLUTELY NO WARRANTY; - [INFO]     .___! |      see `cylc warranty`.  It is free software, you - [INFO]     !_____!       are welcome to redistribute it under certain + [INFO] ._____._. ._| |_____.      Copyright (C) 2008-2019 NIWA + [INFO] | .___| | | | | .___| & British Crown (Met Office) & Contributors. + [INFO] | !___| !_! | | !___. _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ + [INFO] !_____!___. |_!_____! This program comes with ABSOLUTELY NO WARRANTY; + [INFO]     .___! |      see `cylc warranty`.  It is free software, you + [INFO]     !_____!       are welcome to redistribute it under certain [INFO] - [INFO] *** listening on https://<gadi-cpu>.gadi.nci.org.au:<port>/ *** + [INFO] *** listening on https://<persistent-session-full-name>:<port>/ *** [INFO] [INFO] To view suite server program contact information: [INFO] $ cylc get-suite-contact <suite-ID> [INFO] [INFO] Other ways to see if the suite is still running: - [INFO] $ cylc scan -n '<suite-ID>' <gadi-cpu>.gadi.nci.org.au - [INFO] $ cylc ping -v --host=<gadi-cpu>.nci.org.au <suite-ID> - [INFO] $ ps -opid,args <PID> # on <gadi-cpu>.nci.org.au - Cylc GUI + [INFO] $ cylc scan -n '<suite-ID>' <persistent-session-full-name> + [INFO] $ cylc ping -v --host=<persistent-session-full-name> <suite-ID> + [INFO] $ ps -opid,args <PID> # on <persistent-session-full-name> + Cylc GUI --> +
      After running the command rose suite-run, if you get an error similar to the following: 
      [FAIL] Suite "<suite-ID>" appears to be running:
 [FAIL] Contact info from: "/home/565/<$USER>/cylc-run/<suite-ID>/.service/contact"
-[FAIL]    CYLC_SUITE_HOST=gadi.nci.org.au
+[FAIL]    CYLC_SUITE_HOST=<persistent-session-full-name>
 [FAIL]    CYLC_SUITE_OWNER=<$USER>
 [FAIL]    CYLC_SUITE_PORT=<port>
-[FAIL]    CYLC_SUITE_PROCESS=<PID> python2 /usr/local/cylc/cylc-7.8.3/bin/cylc-run <suite-ID>
+[FAIL]    CYLC_SUITE_PROCESS=<PID> /g/data/hr22/apps/cylc7/bin/python -s /g/data/hr22/apps/cylc7/cylc_7.9.7/bin/cylc-run <suite-ID> --host=localhost
 [FAIL] Try "cylc stop '<suite-ID>'" first?
      you should run: 
      rm /home/565/<$USER>/cylc-run/<suite-ID>/.service/contact
      @@ -429,6 +510,10 @@ There are two main ways to restart a suite: [INFO] delete: log/rose-suite-run.version [INFO] symlink: rose-conf/<timestamp>-restart.version <= log/rose-suite-run.version [INFO] chdir: log/ + [WARN] Using the cylc session <persistent-session-full-name> + [WARN] + [WARN] Loading cylc7/23.09 + [WARN]  Loading requirement: mosrs-setup/1.0.1 [INFO]         ._. [INFO]         | |         The Cylc Suite Engine [7.9.7] [INFO] ._____._. ._| |_____.      Copyright (C) 2008-2019 NIWA @@ -438,15 +523,15 @@ There are two main ways to restart a suite: [INFO]     .___! |      see `cylc warranty`.  It is free software, you [INFO]     !_____!       are welcome to redistribute it under certain [INFO] - [INFO] *** listening on https://<gadi-cpu>.nci.org.au:<port>/ *** + [INFO] *** listening on https://<persistent-session-full-name>:<port>/ *** [INFO] [INFO] To view suite server program contact information: [INFO] $ cylc get-suite-contact <suite-ID> [INFO] [INFO] Other ways to see if the suite is still running: - [INFO] $ cylc scan -n '<suite-ID>' <gadi-cpu>.nci.org.au - [INFO] $ cylc ping -v --host=<gadi-cpu>.nci.org.au <suite-ID> - [INFO] $ ps -opid,args <PID> # on <gadi-cpu>.nci.org.au + [INFO] $ cylc scan -n '<suite-ID>' <persistent-session-full-name> + [INFO] $ cylc ping -v --host=<persistent-session-full-name> <suite-ID> + [INFO] $ ps -opid,args <PID> # on <persistent-session-full-name> Cylc GUI
      • @@ -526,6 +611,52 @@ For the u-cy339 suite in this example, the atm directo Files formatted as <suite-name>a.xhist-<year><month><day> contain metadata information. +## Port suites from accessdev +accessdev was the server used for {{ model }} run submission workflow before the update to persistent sessions. +
      +If you have a suite that was running on accessdev, you can run it on persistent sessions by doing the following steps: + +
        +
      1. + Initialisation step +
        + To set the correct SSH configuration for Cylc, some SSH keys need to be created in the ~/.ssh directory. +
        + To create the needed SSH keys, run the following command: + 
        /g/data/hr22/bin/gadi-cylc-setup-ps -y
        +
        + You only need to run this initialisation step once. +
        +
        2. +
      2. + Set host to localhost +
        + To enable Cylc to submit PBS jobs directly from the persistent session, the suite configuration should have its host set as localhost. +
        + You can manually set all occurrencies of host to localhost in the suite configuration files. +
        + Alternatively, you can run the following command in the suite folder: + 
        grep -rl --exclude-dir=.svn "host\s*=" . | xargs sed -i 's/\(host\s*=\s*\).*/\1localhost/g'
        +
        3. +
      3. + Add gdata/hr22 and gdata/ki32 in the PBS storage directives +
        + The persistent sessions workflow uses files in the hr22 and ki32 project folders on Gadi. +
        + Therefore, the respective folders need to be added to the storage directive in the suite configuration files. +
        + You can do this manually, or run the following command in the suite folder: + 
        grep -rl --exclude-dir=.svn "\-l\s*storage\s*=" . | xargs sed -i '/\-l\s*storage\s*=\s*.*gdata\/hr22.*/! s/\(\-l\s*storage\s*=\s*.*\)/\1+gdata\/hr22/g ; /\-l\s*storage\s*=\s*.*gdata\/ki32.*/! s/\(\-l\s*storage\s*=\s*.*\)/\1+gdata\/ki32/g'
        +
        4. +
      + + +
      + Some suites might not be ported this way. +
      + If you have a suite that was running on accessdev and, even after following the steps above, the run submission fails, consider getting help on the Hive Forum. +
      +
      References
      \ No newline at end of file From 01476c3bf04e8e711a52e7a0370eccca1de9fd31 Mon Sep 17 00:00:00 2001 From: Davide Marchegiani Date: Wed, 31 Jan 2024 09:44:34 +1100 Subject: [PATCH 15/17] Update run-access-cm.md Updates after Heidi's review --- docs/models/run-a-model/run-access-cm.md | 28 ++++++++++++------------ 1 file changed, 14 insertions(+), 14 deletions(-) diff --git a/docs/models/run-a-model/run-access-cm.md b/docs/models/run-a-model/run-access-cm.md index fb2f69e31..ccfadd66b 100644 --- a/docs/models/run-a-model/run-access-cm.md +++ b/docs/models/run-a-model/run-access-cm.md @@ -90,7 +90,7 @@ To open the terminal, click on the black terminal icon at the top of the window. Open ARE VDI terminal ## Setup {{ model }} persistent session -To support the use of long-running processes (such as ACCESS models runs), NCI provides a service on Gadi called persistent sessions. +To support the use of long-running processes, such as ACCESS models runs, NCI provides a service on Gadi called persistent sessions. To run {{ model }}, you need to start a persistent session and set it as the target session for the model run. @@ -104,7 +104,7 @@ If you want to assign a different project to the persistent session, use the opt 
      persistent-sessions start -p <project> <name>
      - The project assigned to a persistent session does not have to be necessarily the same to the project used to run {{ model }} configuration, but needs to have allocated Service Units (SU). + While the project assigned to a persistent session does not have to be the same as the project used to run the {{ model }} configuration, it does need to have allocated Service Units (SU).
      For more information, check how to Join relevant NCI projects.
      @@ -114,7 +114,7 @@ If you want to assign a different project to the persistent session, use the opt  ssh <name>.<$USER>.<project>.ps.gadi.nci.org.au -To list all the active persistent sessions run: +To list all active persistent sessions run: 
      persistent-sessions list
      @@ -124,7 +124,7 @@ To list all the active persistent sessions run: -The full name of a newly created persistent session is formatted as: +The label of a newly-created persistent session has the following format:
      <name>.<$USER>.<project>.ps.gadi.nci.org.au. @@ -132,23 +132,23 @@ The full name of a newly created persistent session is formatted as: After starting the persistent session, it is essential to assign it to the {{ model }} run.
      -The easiest way to do so, is to insert the persistent session full name into the file ~/.persistent-sessions/cylc-session. +The easiest way to do this, is to insert the persistent session label into the file ~/.persistent-sessions/cylc-session.
      You can do it manually, or by running the following command: 
      cat > ~/.persistent-sessions/cylc-session <<< <name>.<$USER>.<project>.ps.gadi.nci.org.au
      -For example, if the user dm5220 started a persistent session named cylc, under the project tm70, the command will be: +For example, if the user abc123 started a persistent session named cylc, under the project xy00, the command will be: - cat > ~/.persistent-sessions/cylc-session <<< cylc.dm5220.tm70.ps.gadi.nci.org.au + cat > ~/.persistent-sessions/cylc-session <<< cylc.abc123.xy00.ps.gadi.nci.org.au cat ~/.persistent-sessions/cylc-session - cylc.dm5220.tm70.ps.gadi.nci.org.au + cylc.abc123.xy00.ps.gadi.nci.org.au -For more information on how to specify the target session, check Specify Target Session on Cylc7 Suites. +For more information on how to specify the target session, check Specify Target Session with Cylc7 Suites.
      - You can concurrently submit multiple {{ model }} runs using the same persistent session, without the need to start another one. Therefore, the process of specifying the target persistent session for {{ model }} can potentially be done only once. + You can simultaneously submit multiple {{ model }} runs using the same persistent session without needing to start a new one. Hence, the process of specifying the target persistent session for {{ model }} should only need to be done once.
      After specifying the {{ model }} target persistent session the first time, to run {{ model }} you just have to make sure to have an active persistent session named like the {{ model }} target persistent session.
      @@ -191,9 +191,9 @@ module load cylc7/23.09
      - Make sure to load a version of Cylc >= 23.09, as earlier versions do not support the persistent sessions workflow. + Make sure to load a version of Cylc >= 23.09 as earlier versions do not support the persistent sessions workflow.
      - Also, before loading the Cylc module, make sure to have started a persistent session and to have assigned it to the {{ model }} workflow. For more information about these steps, check Setup {{ model }} persistent session. + Also, before loading the Cylc module, make sure to have started a persistent session and assigned it to the {{ model }} workflow. For more information about these steps, check Setup {{ model }} persistent session.
    • MOSRS authentication @@ -614,7 +614,7 @@ Files formatted as <suite-name>a.xhist-<year><month>< ## Port suites from accessdev accessdev was the server used for {{ model }} run submission workflow before the update to persistent sessions.
      -If you have a suite that was running on accessdev, you can run it on persistent sessions by doing the following steps: +If you have a suite that was running on accessdev, you can run it using persistent sessions by carrying out the following steps:
      1. @@ -678,4 +678,4 @@ If you have a suite that was running on accessdev, you can run it on pers
      2. https://opus.nci.org.au/display/Help/Persistent+Sessions
        3. -
    \ No newline at end of file + From f341f91eee0d3459140e4af85ea57fa9336e63b2 Mon Sep 17 00:00:00 2001 From: atteggiani Date: Mon, 12 Feb 2024 14:52:55 +1100 Subject: [PATCH 16/17] Edits after Heidi's reviews --- docs/models/run-a-model/run-access-cm.md | 22 ++++++++++++---------- 1 file changed, 12 insertions(+), 10 deletions(-) diff --git a/docs/models/run-a-model/run-access-cm.md b/docs/models/run-a-model/run-access-cm.md index ccfadd66b..7ff63b6b9 100644 --- a/docs/models/run-a-model/run-access-cm.md +++ b/docs/models/run-a-model/run-access-cm.md @@ -3,7 +3,9 @@
    Important for accessdev users!
    - If you are an accessdev user, make sure you are a member of hr22 and ki32 projects, and then check the new persistent session worflow for {{ model }}, and how to port suites from accessdev. + If you are an accessdev user, make sure you are a member of hr22 and ki32 projects. +
    + Then, refer to instructions on how to set up persistent session worflow for {{ model }}, and how to port suites from accessdev.
    ## Prerequisites ### General prerequisites @@ -32,18 +34,18 @@ If you are unsure whether {{ model }} is the right choice for your experiment, t
  • Connection to an ARE VDI Desktop (optional)
    - To run {{ model }}, it is preferred to start an Australian Research Environment (ARE) VDI Desktop session. + To run {{ model }}, start an Australian Research Environment (ARE) VDI Desktop session.
    If you are not familiar with ARE, check out the Getting Started on ARE section.
    • -------------------------------------------- -## Setup ARE VDI Desktop (optional) -To skip this step and run {{ model }} from Gadi login node instead, check Setup {{ model }} persistent session. +## Set up an ARE VDI Desktop (optional) +To skip this step and instead run {{ model }} from Gadi login node, refer to instructions on how to set up {{ model }} persistent session. ### Launch ARE VDI Session -Go to the ARE VDI page and launch a session with the following directives: +Go to the ARE VDI page and launch a session with the following entries:
    • Walltime (hours) → 2 @@ -60,7 +62,7 @@ Go to the -## Setup {{ model }} persistent session -To support the use of long-running processes, such as ACCESS models runs, NCI provides a service on Gadi called persistent sessions. +## Set up {{ model }} persistent session +To support the use of long-running processes, such as ACCESS model runs, NCI provides a service on Gadi called persistent sessions. To run {{ model }}, you need to start a persistent session and set it as the target session for the model run. ### Start a new persistent session -To start a new persistent session on Gadi (in a login node, or in an ARE terminal instance), run the following command: +To start a new persistent session on Gadi, using either a login node or an ARE terminal instance, run the following command: 
      persistent-sessions start <name>
      This will start a persistent session with the given name that runs under your default project. @@ -193,7 +195,7 @@ module load cylc7/23.09
      Make sure to load a version of Cylc >= 23.09 as earlier versions do not support the persistent sessions workflow.
      - Also, before loading the Cylc module, make sure to have started a persistent session and assigned it to the {{ model }} workflow. For more information about these steps, check Setup {{ model }} persistent session. + Also, before loading the Cylc module, make sure to have started a persistent session and assigned it to the {{ model }} workflow. For more information about these steps, check Set up {{ model }} persistent session.
    • MOSRS authentication From 5bbb1b7072055600a5d7075356907b4146232c88 Mon Sep 17 00:00:00 2001 From: atteggiani Date: Mon, 12 Feb 2024 16:40:06 +1100 Subject: [PATCH 17/17] Updates after Heidi's review --- docs/models/run-a-model/run-access-cm.md | 28 +++++++++++------------- 1 file changed, 13 insertions(+), 15 deletions(-) diff --git a/docs/models/run-a-model/run-access-cm.md b/docs/models/run-a-model/run-access-cm.md index 7ff63b6b9..00a0fd4f8 100644 --- a/docs/models/run-a-model/run-access-cm.md +++ b/docs/models/run-a-model/run-access-cm.md @@ -5,7 +5,7 @@
      If you are an accessdev user, make sure you are a member of hr22 and ki32 projects.
      - Then, refer to instructions on how to set up persistent session worflow for {{ model }}, and how to port suites from accessdev. + Then, refer to instructions on how to Set up persistent session worflow for {{ model }}, and how to port suites from accessdev. ## Prerequisites ### General prerequisites @@ -42,7 +42,7 @@ If you are unsure whether {{ model }} is the right choice for your experiment, t -------------------------------------------- ## Set up an ARE VDI Desktop (optional) -To skip this step and instead run {{ model }} from Gadi login node, refer to instructions on how to set up {{ model }} persistent session. +To skip this step and instead run {{ model }} from Gadi login node, refer to instructions on how to Set up {{ model }} persistent session. ### Launch ARE VDI Session Go to the ARE VDI page and launch a session with the following entries: @@ -134,13 +134,13 @@ The label of a newly-created persistent session has the following format: After starting the persistent session, it is essential to assign it to the {{ model }} run.
      -The easiest way to do this, is to insert the persistent session label into the file ~/.persistent-sessions/cylc-session. +The easiest way to do this is to insert the persistent session label into the file ~/.persistent-sessions/cylc-session.
      You can do it manually, or by running the following command: 
      cat > ~/.persistent-sessions/cylc-session <<< <name>.<$USER>.<project>.ps.gadi.nci.org.au
      -For example, if the user abc123 started a persistent session named cylc, under the project xy00, the command will be: +For example, if the user abc123 started a persistent session named cylc under the project xy00, the command will be: cat > ~/.persistent-sessions/cylc-session <<< cylc.abc123.xy00.ps.gadi.nci.org.au @@ -148,11 +148,11 @@ For example, if the user abc123 started a persistent session named cylc.abc123.xy00.ps.gadi.nci.org.au -For more information on how to specify the target session, check Specify Target Session with Cylc7 Suites. +For more information on how to specify the target session, refer to Specify Target Session with Cylc7 Suites.
      You can simultaneously submit multiple {{ model }} runs using the same persistent session without needing to start a new one. Hence, the process of specifying the target persistent session for {{ model }} should only need to be done once.
      - After specifying the {{ model }} target persistent session the first time, to run {{ model }} you just have to make sure to have an active persistent session named like the {{ model }} target persistent session. + After specifying the {{ model }} target persistent session the first time, to run {{ model }} you just need to make sure to have an active persistent session named like the {{ model }} target persistent session.
      ### Terminate a persistent session @@ -168,7 +168,7 @@ To stop a persistent session, run:
      Each {{ model }} suite has a suite-ID in the format u-<suite-name>, where <suite-name> is a unique identifier.
      -For this example you can use u-cy339, which is a preindustrial experiment suite. +For this example you can use u-cy339, which is a pre-industrial experiment suite.
      Typically, an existing suite is copied and then edited as needed for a particular run. @@ -193,9 +193,9 @@ module load cylc7/23.09
      • - Make sure to load a version of Cylc >= 23.09 as earlier versions do not support the persistent sessions workflow. + Make sure to load Cylc version 23.09 (or later), as earlier versions do not support the persistent sessions workflow.
      - Also, before loading the Cylc module, make sure to have started a persistent session and assigned it to the {{ model }} workflow. For more information about these steps, check Set up {{ model }} persistent session. + Also, before loading the Cylc module, make sure to have started a persistent session and assigned it to the {{ model }} workflow. For more information about these steps, refer to instructions on how to Set up {{ model }} persistent session.
    • MOSRS authentication @@ -624,7 +624,7 @@ If you have a suite that was running on accessdev, you can run it using persiste
      To set the correct SSH configuration for Cylc, some SSH keys need to be created in the ~/.ssh directory.
      - To create the needed SSH keys, run the following command: + To create the required SSH keys, run the following command: 
      /g/data/hr22/bin/gadi-cylc-setup-ps -y
      You only need to run this initialisation step once. @@ -635,7 +635,7 @@ If you have a suite that was running on accessdev, you can run it using persiste
      To enable Cylc to submit PBS jobs directly from the persistent session, the suite configuration should have its host set as localhost.
      - You can manually set all occurrencies of host to localhost in the suite configuration files. + You can manually set all occurrences of host to localhost in the suite configuration files.
      Alternatively, you can run the following command in the suite folder: 
      grep -rl --exclude-dir=.svn "host\s*=" . | xargs sed -i 's/\(host\s*=\s*\).*/\1localhost/g'
      @@ -643,11 +643,9 @@ If you have a suite that was running on accessdev, you can run it using persiste
    • Add gdata/hr22 and gdata/ki32 in the PBS storage directives
      - The persistent sessions workflow uses files in the hr22 and ki32 project folders on Gadi. + As the persistent sessions workflow uses files in the hr22 and ki32 project folders on Gadi, the respective folders need to be added to the storage directive in the suite configuration files.
      - Therefore, the respective folders need to be added to the storage directive in the suite configuration files. -
      - You can do this manually, or run the following command in the suite folder: + You can do this manually or run the following command from within the suite directory: 
      grep -rl --exclude-dir=.svn "\-l\s*storage\s*=" . | xargs sed -i '/\-l\s*storage\s*=\s*.*gdata\/hr22.*/! s/\(\-l\s*storage\s*=\s*.*\)/\1+gdata\/hr22/g ; /\-l\s*storage\s*=\s*.*gdata\/ki32.*/! s/\(\-l\s*storage\s*=\s*.*\)/\1+gdata\/ki32/g'