Skip to content

Commit

Permalink
Merge pull request #548 from ACCESS-Hive/dev/heidi/issue546
Browse files Browse the repository at this point in the history
Thank you @heidinett
  • Loading branch information
atteggiani authored Aug 29, 2023
2 parents 15f55f0 + cc0b81b commit f8e5145
Show file tree
Hide file tree
Showing 9 changed files with 150 additions and 92 deletions.
Binary file removed docs/assets/model-config-logos/access-am-config.png
Binary file not shown.
Binary file not shown.
Binary file not shown.
12 changes: 6 additions & 6 deletions docs/model_evaluation/index.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,16 +2,16 @@

<!-- Model evaluation is about measuring how fit for purpose a particular model is. -->

Model evaluation in climate science is the process of assessing the performance and reliability of computational models that simulate Earth's climate system. It involves comparing model predictions to observed data to determine the model's accuracy and usefulness. This process helps understand how well a model represents real-world climate processes and predict future climate trends. Through rigorous evaluation, scientists can identify model strengths, weaknesses, and uncertainties, refining models to enhance their predictive capabilities.
Model evaluation in climate science is the process of assessing the performance and reliability of computational models that simulate the Earth's climate system. It involves comparing model predictions to observed data to determine the model's accuracy and usefulness. In doing so, we can understand how well a model represents real-world climate processes and make predictions about future climate trends. Such rigorous model evaluation allows scientists to identify model strengths, weaknesses and uncertainties, as well as refine models to enhance their predictive capabilities.

## Getting Started with MED

If you are new to MED and are wondering [*"What is Model Evaluation and Diagnostics about?"*](./model_evaluation_getting_started/index.md), we recommend you read our [Getting Started with MED page](./model_evaluation_getting_started/index.md):
<!--If you are new to MED and are wondering [*"What is Model Evaluation and Diagnostics about?"*](./model_evaluation_getting_started/index.md), we recommend you read our [Getting Started with MED page](./model_evaluation_getting_started/index.md): -->

<div class="card-container">
<a href="../getting_started" class="vertical-card aspect-ratio1to1">
<div class="card-image-container">
<img src="..//assets/model_evaluation/Gadi-19-2.jpg" alt="Computing Access" class="img-cover"></img>
<div class="vertical-card-image-container">
<img src="../assets/model_evaluation/Gadi-19-2.jpg" alt="Computing Access" class="img-cover"></img>
</div>
<div class="card-text-container bold ">Computing Access</div>
</a>
Expand Down Expand Up @@ -89,12 +89,12 @@ If you are new to MED and are wondering [*"What is Model Evaluation and Diagnost

### Tools in development

We are currently setting up a range of tools that will help you to better evaluate and diagnose climate models:
For the evaluation and diagnosis of ACCESS climate models, the following tools are currently being setup:

* Data format processing tools like APP4
* An Evaluation Recipe Gallery with searching functionality

While we are working on these, we have collected a number of links to existing tools in our [community tab](../community_resources/index.md) (note that we are not currating them).
While this work is in progress, you can refer to a collection of links to existing tools (not curated by ACCESS-NRI) in the [community tab](../community_resources/index.md).

<!-- {% include "call_contribute.md" %} -->

Expand Down
21 changes: 15 additions & 6 deletions docs/model_evaluation/model_evaluation_getting_started/index.md
Original file line number Diff line number Diff line change
@@ -1,16 +1,25 @@
# Getting Started with Model Evaluation at NCI

Welcome to Model Evaluation and Diagnostics!
<!--Welcome to Model Evaluation and Diagnostics! -->

## What is MED about?
## What is MED?

**Evaluation** involves scrutinizing the model through Model/Observation confrontations, checking its performance against real-world observations. It also includes experiment comparisons, testing the model under different scenarios, and inter-model comparisons like the Coupled Model Intercomparison Project (CMIP), assessing how the ACCESS-NRI model fares when compared with other climate models.
Model Evaluation and Diagnostics (MED) of ACCESS models includes:

**Diagnostics** involves constant monitoring of model runs to detect any anomalies or inconsistencies and a thorough analysis of outputs to verify the model's accuracy over time.
<ul>
<li><b>Evaluation</b>
<br>
Model evaluation involves scrutinising a model's performance by comparing its output with observations and through Model/Observation confrontations. It also includes experiment comparisons, model testing with different scenarios and inter-model comparisons such as the Coupled Model Intercomparison Project (CMIP).
<li>
<b>Diagnostics</b>
<br>
Model diagnostics is essentially constant monitoring of model runs to detect any anomalies or inconsistencies. Thorough analysis of outputs are used to verify the model's accuracy over time.
<br>
</ul>

## How can you get started with MED?
## How to get started with MED?

Here, we provide you the important information to give you access to the large data that we curate at NCI's storage and show you how you can use it to figure out how fit for purpose specific models are, in particular when you compare them to osbervational data:
Here you can find information on how to access curated data stored at NCI and how to use it to evaluate specific models, such as comparing model output with observational data.

<div class="card-container">
<a href="../../getting_started" class="vertical-card aspect-ratio1to1">
Expand Down
Original file line number Diff line number Diff line change
@@ -1,30 +1,37 @@
# `conda` Environment for Model Evaluation on NIC's Gadi
# `conda` Environment for Model Evaluation on Gadi

At this stage of *Getting Started*, we assume that you already have access to NCI's Gadi via `ssh`. If this is not the case, please go to our instructions on [how to get access to NCI's Gadi](../../getting_started/index.md).
If you do not yet have `ssh` access to <i>Gadi</i>, refer to instructions on how to <a href="/getting_started#login-to-gadi">login to Gadi</a>.

The instructions below explain how to load our curated `python` environment, with packages and scripts which are supported by ACCESS-NRI. Once these instructions have been followed you will be able to use all pacakges and scripts when running directly on Gadi via `ssh`, in `PBS` scripts, or in JupyterLab.
The following instructions explain how to load the curated `python` environment on NCI, which includes packages and scripts supported by ACCESS-NRI. Once loaded, these can be run directly on <i>Gadi</i> via `ssh`, `PBS` scripts, or in `JupyterLab`.

???+ warning "ACCESS-NRI provides code and support, but not computing resources"
As mentioned in the [Getting Started pages](../../../getting_started), you do not automatically have access to all of Gadi's storage at `/g/data/`, but need to be part of a `$PROJECT` to see files at `/g/data/$PROJECT`. For model evaluation and diagnostics, you need to be part of projects `xp65` and `hh5` for code access and a project with compute resources.
???+ warning "ACCESS-NRI can provide code and support, but not computing resources"
You do not automatically have access to all `/g/data/` storage on <i>Gadi</i>. You need to <a href="/getting_started/first_steps#join-relevant-nci-projects">join an NCI project</a> to view files on `/g/data/$PROJECT`.
<br>
For model evaluation and diagnostics, you need to join projects `xp65` and `hh5` for code access and a `$PROJECT` with sufficient compute resources.

## What is part of the `access-med` enrivonment?
## What is the `access-med` environment?

The complete list of dependencies can be found in <a href="https://github.com/ACCESS-NRI/MED-condaenv/blob/main/scripts/environment.yml" target="_blank">this</a> `environment.yml` file of our <a href="https://github.com/ACCESS-NRI/MED-condaenv" target="_blank">GitHub repository</a> and includes `intake`, `esmvaltool`, and `ilamb`:
The complete list of dependencies for the `access-med` environment can be found in the <a href="https://github.com/ACCESS-NRI/MED-condaenv/blob/main/scripts/environment.yml" target="_blank">environment.yml</a> file of the <a href="https://github.com/ACCESS-NRI/MED-condaenv" target="_blank">ACCESS-NRI MED GitHub repository</a>. These include `intake`, `esmvaltool` and `ilamb`:
<div style="text-align: center;">
<img src="../../../assets/model_evaluation/condaenv_list.png" alt="List of packages that are provided as part of the xp65 access-med environment" width="75%"/>
</div>

## Running our `access-med` environment on Gadi
## Running the `access-med` environment on Gadi

To avoid running code on Gadi with incompatible packages we provide you with a conda environment called access-med.
In order to change to this curated environment, please run the following commands everytime after you log into Gadi (and as part of a <a href="https://opus.nci.org.au/display/Help/PBS+Directives+Explained" target="_blank">PBS scripts</a>):
To avoid running code on <i>Gadi</i> with incompatible packages, a conda environment called `access-med` is provided.
<br>
To change to this curated environment, run the following commands after logging into <i>Gadi</i> and <a href="https://opus.nci.org.au/display/Help/PBS+Directives+Explained" target="_blank">edit</a> your `PBS` script accordingly:
```
module use /g/data/xp65/public/modules
module load conda/access-med
```

This will load the latest version of `access-med` (in this tutorial `0.3`). To check which conda version you are using, you can run `which python`:

This will load the latest version of `access-med`, e.g. version `access-med-0.3`.
<br>
To check which `conda` version you are using, run the following command:
```
which python
```

<terminal-window>
<terminal-line data="input">module use /g/data/xp65/public/modules</terminal-line>
Expand All @@ -36,7 +43,7 @@ This will load the latest version of `access-med` (in this tutorial `0.3`). To c
<terminal-line>/g/data/xp65/public/apps/med_conda_scripts/access-med-0.3.d/bin/python</terminal-line>
</terminal-window>

To test everything is working correctly, import the packages in `python3`:
To test everything is working correctly, import the packages in `python3` as follows:

```python
import numpy as np
Expand All @@ -49,13 +56,7 @@ print(intake.__version__)
print(esmvaltool.__version__)
```

If you are planning to run your code on Gadi with a Portable Batch System (`PBS`) job, you will need to add in the `module use` and `module load` commands to your PBS script as well.

<div class="note">
If you are not familiar with PBS jobs on NCI, you can find NCI's guide <a href="https://opus.nci.org.au/display/Help/4.+PBS+Jobs" target="_blank">here</a>.
</div>

You could for example create an `example_pbs.sh` file with the content:
If you want to run your code on <i>Gadi</i> using a Portable Batch System (`PBS`) job, add the `module use` and `module load` commands to your `PBS` script as shown in the `example_pbs.sh` `PBS` script below:

```
#!/bin/bash
Expand All @@ -74,44 +75,61 @@ module load conda/access-med
python3 your_code.py
```

The content of `your_code.py` could be as simple as the import and version print from our example above. submit this job, you then only need to execute
The content of `your_code.py` could simply comprise the `import` and `which version` lines from our above example.
<br>
To submit this `PBS` job, execute the following command:
```
qsub example_pbs.sh
```

In brief: this PBS script will submit a job to Gadi with the job name (`#PBS -N`) *example_pbs* under compute project (`#PBS -P`) `iq82` with a <a href="https://opus.nci.org.au/display/Help/Queue+Limits" target="_blank">normal queue</a> (`#PBS -q normalbw`), for 1 CPU (`#PBS -l ncpus=1`) with 2 GB RAM (`#PBS -l mem=2GB`), a walltime of 10 minutes (`#PBS -l walltime=00:10:00`) and data storage access to projects `xp65`. Note that for this example to work, you have to be <a href="https://my.nci.org.au/mancini/project-search" target="_blank">member of the NCI project</a> `xp65` and `iq82`. Adjust the `#PBS -P` option to match your compute project. Upon starting the job, it will change into to the working directory that you submitted the job from (`#PBS -l wd`) and load the access-med conda environment.


## Running our `access-med` environment on NCI's Interactive ARE (JupyterLab)

NCI also supports an interactive coding environment called the Australian Research Environment (ARE). It's use is quite similar to submitting a computing job via `qsub -I`, but it comes with dedicated graphical user interfaces for jupyter notebooks. To use it, you need an NCI account and be part of a project that gives you computing resources (see our [getting started](../../getting_started/index.md)).

You can then go to <a href="https://are.nci.org.au" target="_blank">https://are.nci.org.au</a> to log in. In the "Featured Apps" section, click on "JupyterLab" and to a JupyterLab instance.

<div class="note">
If you are not familiar with ARE at NCI, you can find NCI's guide <a href="https://opus.nci.org.au/display/Help/3.1+Starting+JupyterLab+App" target="_blank">here</a>.
</div>

Below we have provided example values, however you must change these values to match your project and use case:
This will submit a job to <i>Gadi</i> with the job name (`#PBS -N`) *example_pbs* under compute project (`#PBS -P`) *iq82* with a </i>normalbw</i> <a href="https://opus.nci.org.au/display/Help/Queue+Limits" target="_blank">normal queue</a> (`#PBS -q`). The </i>number of CPUs</i> requested is 1 CPU (`#PBS -l ncpus=1`) with 2 GB RAM (`#PBS -l mem=2GB`) and a <i>walltime</i> of 10 minutes (`#PBS -l walltime=00:10:00`). The <i>data storage</i> (`#PBS -l storage=gdata/xp65`) is data storage access to project `xp65`.
<br>
<br>
<i>Note</i>: to run this example, you need to be a <a href="https://my.nci.org.au/mancini/project-search" target="_blank">member of an NCI project</a>, in this case `xp65` and `iq82` projects.
<br>
Adjust the `#PBS -P` option to match your compute project.
<br>
When the job starts, it will change to the working directory from where you submitted the job (`#PBS -l wd`) and load the access-med `conda` environment.
<br>
<br>
For more information on running `PBS` jobs on NCI, refer to <a href="https://opus.nci.org.au/display/Help/4.+PBS+Jobs" target="_blank">PBS Jobs</a>.

## Running the `access-med` environment on ARE

NCI also supports an interactive coding environment called the Australian Research Environment (<i>ARE</i>). Its use is similar to that for submitting a `PBS` job via `qsub -I`, but with an added bonus of a dedicated graphical user interface for `Jupyter` notebooks.
<br>
<br>
To use <i>ARE</i>, you must have an NCI account and be a member of a project with computing resources (see section on [getting started](../../getting_started/index.md)).

Once you <a href="https://are.nci.org.au" target="_blank">login to <i>ARE</i></a>, click on <i>JupyterLab</i> in the <i>Featured Apps</i> section to launch a `JupyterLab` instance.
<br>
Below are some example values that you should change to match your `$PROJECT` and use case:

- **Walltime (hours)** `1`
- **Queue** `normalbw`
- **Compute Size** `tiny`
- **Project** `iq82` (This should match your project with computing resources)
- **Storage** `gdata/xp65+gdata/hh5` (Select all which match your project's gdata storage)
- **Project** `iq82` (This should match your `$PROJECT` with compute resources)
- **Storage** `gdata/xp65+gdata/hh5` (Select all that match your project's `/g/data` storage)
- *Advanced Options ...* (click button to expand)
- **Module directories** `/g/data/xp65/public/modules`
- **Modules** `conda/are`
- *Launch* (click to submit)

This will launch a JupyterLab session with a Session ID and display it in the list of interactive sessions (you can also find it under **My Interactive Sessions** in the top left of the ARE website).
The session appears blue while it is loading, yellow or red in case of warnings or errors, and green when it is successfully running, as in the following example:
This will launch a `JupyterLab` session with a <i>Session ID</i>, which will appear in the list of interactive sessions. (You can also find it under <i>My Interactive Sessions</i> at the top-left of the ARE window).
<br>
The session appears blue while it is loading, yellow or red in case of warnings or errors, and green when it is successfully running:

<div style="text-align: center;">
<img src="../../../assets/getting_started/are_1.png" alt="Example of a successfully started ARE Session" width="75%"/>
</div>

You can then **Open JupyterLab** via a button at the bottom of the session. This will bring you to a window with a directory structure to the left and a jupyter notebook to the right (see the example below). If you have loaded the modules from `hh5` or `xp65`, you should then be able to import python packes like `numpy`, `xarray` or `intake`, as shown in the screenshot below.
You can then <i>Open JupyterLab</i> by clicking on the button at the bottom of the session.
<br>
This will open a window which contains a directory structure on the left and a `Jupyter` notebook on the right, as shown below.
<br>
If you loaded the modules from `hh5` or `xp65`, you should be able to import python packages such as `numpy`, `xarray` or `intake`, as shown below:

<div style="text-align: center;">
<img src="../../../assets/getting_started/are_2.png" alt="Example of a JupyterLab session with directory tree to the left and jupyter notebook to the right, showing successfully imported python packages." width="75%" />
Expand Down
Loading

0 comments on commit f8e5145

Please sign in to comment.