Skip to content

Commit

Permalink
edited ESMValTool section
Browse files Browse the repository at this point in the history
  • Loading branch information
@heidinett
heidinett committed Aug 31, 2023
1 parent 98bc414 commit 7f19d17
Show file tree
Hide file tree
Showing 2 changed files with 62 additions and 69 deletions.
93 changes: 46 additions & 47 deletions ...odel_evaluation/model_evaluation_on_gadi/model_evaluation_on_gadi_esmvaltool.md
View file
Original file line number Diff line number Diff line change
@@ -1,13 +1,16 @@
# Tutorial for using `esmvaltool` on Gadi@NCI
# Tutorial for using ESMValTool on Gadi

`esmvaltool` is the Earth System Model Evaluation Tool.
## What is ESMValTool
The Earth System Model Evaluation Tool (<i>ESMValTool</i>) is a climate model diagnostics and evaluation software package to better understand the causes and effects of model biases and inter-model spread.

???+ warning "Support Level: Supported on Gadi, but not owned by ACCESS-NRI"
<!-- Who develped the tool? -->
ESMValTool is a community-developed climate model diagnostics and evaluation software package.
???+ warning "Support Level: Supported on <i>Gadi</i>, but not owned by ACCESS-NRI"
<!-- Code ownership and support -->
ACCESS-NRI does not own the code of ESMValTool, but actively supports the use of ESMValTool on Gadi.
ACCESS-NRI provides access to the latest version of ESMValTool via the `xp65` access-med conda environment deployed on NCI-Gadi.
<i>ESMValTool</i> is a community-developed climate model diagnostics and evaluation software package. While ACCESS-NRI does not own the code, it actively supports the use of <i>ESMValTool</i> software on Gadi. ACCESS-NRI provides access to the latest version of <i>ESMValTool</i> via the `xp65` <i>access-med conda environment</i> deployed on <i>Gadi</i>.


The <i>ESMValTool</i> mainly focuses on evaluating results from the Coupled Model Intercomparison Project (CMIP) ensemble.
<!-- The goal is to build a common framework for the evaluation of Earth System Models (ESMs) against observations available through the Earth System Grid Federation (ESGF) in standard formats (obs4MIPs) or made available at ESGF nodes. -->
For more information, refer to the official <a href="https://docs.esmvaltool.org/en/latest" target="_blank">ESMValTool documentation</a>.

<div class="card-container">
<a href="https://docs.esmvaltool.org/en/latest/" target="_blank" class="vertical-card aspect-ratio2to1">
Expand All @@ -18,74 +21,49 @@
</a>
</div>

## About ESMValTool

The Earth System Model Evaluation Tool (ESMValTool) is a community-development that aims at improving diagnosing and understanding of the causes and effects of model biases and inter-model spread. The ESMValTool mainly focus on evaluating results from the Coupled Model Intercomparison Project (CMIP) ensemble. The goal is to build a common framework for the evaluation of Earth System Models (ESMs) against observations available through the Earth System Grid Federation (ESGF) in standard formats (obs4MIPs) or made available at ESGF nodes.

More information on ESMValTool scope is available in the extensive <a href="https://docs.esmvaltool.org/en/latest" target="_blank">ESMValTool documentation</a>.

ACCESS-NRI provides access to the latest version of ESMValTool via the `xp65` access-med conda environment deployed on NCI-Gadi.
Our plan is to routinely run benchmarks and comparisons of the ACCESS models CMIP submissions. We will also provide support for running recipes on NCI-Gadi.
## Using ESMValTool on Gadi

## Running `esmvaltool` on Gadi
<i>ESMValTool</i> is provided through the `xp65` <i>access-med</i> conda environment deployed on <i>Gadi</i>.
<br>
ACCESS-NRI plans to routinely run benchmarks and comparisons of CMIP submissions for ACCESS models, as well as providing support to run <i>ESMValTool</i> recipes on Gadi.

#### Pre-requisites
### Pre-requisites

NCI Projects requires to run the set of ESMValTool recipes:
To run <i>ESMValTool</i> recipes, you need to be a member of the following NCI projects:

- xp65, ct11, fs38, oi10, rr3, al33, rt52, zz93, qv56


#### Load the `access-med` conda environment
- `xp65`, `ct11`, `fs38`, `oi10`, `rr3`, `al33`, `rt52`, `zz93` and `qv56`.

### Running ESMValTool recipes
<!-- #### Load the `access-med` conda environment -->
To load the the `access-med` conda environment, execute the following commands:
```
module use /g/data/xp65/public/modules
module load conda/access-med
```

#### What recipes are available?
To list which <i>ESMValTool</i> recipes are available on Gadi, run:
```
esmvaltool recipes list
```

#### Details of a recipe
To find out details of a specific `recipe_name.yml`, execute:
```
esmvaltool recipes show recipe_name.yml
```

#### Running an recipe yourself
To execute `recipe_name.yml` and automatically download the required climate data to the default directory, run:

```
esmvaltool run examples/recipe_python.yml --search_esgf=when_missing
```

## Support

ACCESS and NCI-Gadi users can get help from ACCESS-NRI for running their recipe on Gadi via Github Issue on the <a href="https://github.com/ACCESS-NRI/ESMValTool-workflow.git" target="_blank">ESMValTool-Workflow</a> github repository or by opening a thread on the <a href="https://access-hive.org.au" target="_blank">ACCESS-Hive Forum</a>.

General support for ESMValTool (non-specific to NCI-Gadi) can be found in <a href="https://github.com/ESMValGroup/ESMValTool/discussions" target="_blank">ESMValTool Discussions page</a> where users can open an issue and a member of the User Engagement Team of ESMValTool will reply as soon as possible. This is open for all general and technical questions on the ESMValTool: installation, application, development, or any other question or comment you may have.

### Recipes and diagnostics

Contacts for specific diagnostic sets are the respective authors, as listed in the corresponding <a href="https://docs.esmvaltool.org/en/latest/recipes/index.html#recipes" target="_blank">recipe and diagnostic documentation</a> and in the source code.

The current status of ESMValTool recipes for the `xp65` conda environment is available <a href="https://github.com/ACCESS-NRI/ESMValTool-workflow.git" target="_blank">here</a>

## License

The ESMValTool is released under the Apache License, version 2.0. Citation of the ESMValTool paper (“Software Documentation Paper”) is kindly requested upon use, alongside with the software DOI for ESMValTool (doi:10.5281/zenodo.3401363) and ESMValCore (doi:10.5281/zenodo.3387139) and version number:

> Righi, M., Andela, B., Eyring, V., Lauer, A., Predoi, V., Schlund, M., Vegas-Regidor, J., Bock, L., Brötz, B., de Mora, L., Diblen, F., Dreyer, L., Drost, N., Earnshaw, P., Hassler, B., Koldunov, N., Little, B., Loosveldt Tomas, S., and Zimmermann, K.: Earth System Model Evaluation Tool (ESMValTool) v2.0 – technical overview, Geosci. Model Dev., 13, 1179–1199, https://doi.org/10.5194/gmd-13-1179-2020, 2020.
Besides the above citation, users are kindly asked to register any journal articles (or other scientific documents) that use the software at the ESMValTool webpage (http://www.esmvaltool.org/). Citing the Software Documentation Paper and registering your paper(s) will serve to document the scientific impact of the Software, which is of vital importance for securing future funding. You should consider this an obligation if you have taken advantage of the ESMValTool, which represents the end product of considerable effort by the development team.
The `--search_esgf=when_missing` option tells <i>ESMValTool</i> to search for and download the necessary climate data files from Earth System Grid Federation (ESGF), if they cannot be found locally.

## ESMValTool recipe examples

<!-- Explain what the Tiers mean: Tier3 not to be distributed / license issue, Tier2: some restrictions, but can be redistributed while citing papers etc., Tier1: open for everyone -->

To find the available recipes, please go see the <a href="https://github.com/ACCESS-NRI/ESMValTool-workflow.git" target="_blank">ACCESS ESMValTool Worflow recipe status</a>
A list of <i>ESMValTool</i> recipes available on <i>Gadi</i> can be found on the <a href="https://github.com/ACCESS-NRI/ESMValTool-workflow.git" target="_blank">ACCESS-NRI MED ESMValTool Workflow</a> GitHub repository. Some example recipes are provided below:

Below we showcase example recipes from `esmvaltool` that we are providing to run on Gadi:

<!-- Compare to list from https://github.com/ACCESS-NRI/ESMValTool-workflow/issues/103 -->

Expand All @@ -110,6 +88,27 @@ Below we showcase example recipes from `esmvaltool` that we are providing to run
</a>
</div>

## Support

To get help running your <i>ESMValTool</i> recipe on <i>Gadi</i>, you can submit an issue on the <a href="https://github.com/ACCESS-NRI/ESMValTool-workflow.git" target="_blank">ESMValTool-Workflow</a> GitHub repository or ask for help on the <a href="https://access-hive.org.au" target="_blank">ACCESS-Hive Forum</a>.

General <i>ESMValTool</i> support (i.e. non-specific to <i>Gadi</i>) can be found on the <a href="https://github.com/ESMValGroup/ESMValTool/discussions" target="_blank">ESMValTool Discussions</a> page, where users can also post technical questions on the <i>ESMValTool</i> installation, application and development.

### Recipes and diagnostics

Contacts for specific diagnostic sets are the authors listed in the source code and corresponding <a href="https://docs.esmvaltool.org/en/latest/recipes/index.html#recipes" target="_blank">recipe and diagnostic documentation</a>.

The current status of <i>ESMValTool</i> recipes for the `xp65` conda environment on <i>Gadi</i> is available <a href="https://github.com/ACCESS-NRI/ESMValTool-workflow.git" target="_blank">here</a>

### License

The <i>ESMValTool</i> is released under the Apache License, version 2.0. Citation of the <i>ESMValTool</i> paper (“Software Documentation Paper”) is requested upon use, along with the software DOI for <i>ESMValTool</i> (doi:10.5281/zenodo.3401363) and <i>ESMValCore</i> (doi:10.5281/zenodo.3387139) together with the version:

> Righi, M., Andela, B., Eyring, V., Lauer, A., Predoi, V., Schlund, M., Vegas-Regidor, J., Bock, L., Brötz, B., de Mora, L., Diblen, F., Dreyer, L., Drost, N., Earnshaw, P., Hassler, B., Koldunov, N., Little, B., Loosveldt Tomas, S., and Zimmermann, K.: Earth System Model Evaluation Tool (ESMValTool) v2.0 – technical overview, Geosci. Model Dev., 13, 1179–1199, https://doi.org/10.5194/gmd-13-1179-2020, 2020.
Besides the above citation, users are asked to register any journal articles (or other scientific documents) that use the software at the ESMValTool webpage (http://www.esmvaltool.org/). Citing the Software Documentation Paper and registering your paper(s) will serve to document the scientific impact of the Software, which is important for securing future funding. You should consider this an obligation if you have taken advantage of the <i>ESMValTool</i>, which represents the end product of considerable effort by the development team.


<!-- <tr>
<td><a href="/assets/model_evaluation/esmvaltool/fig-9-8.png"><img src="/assets/model_evaluation/esmvaltool/fig-9-8.png" title="Global average 2m temperature anomalies; resembling Flato et al. (2013), Fig. 9.8." /></a></td>
<td><a href="/assets/model_evaluation/esmvaltool/fig-9-4.png"><img src="/assets/model_evaluation/esmvaltool/fig-9-4.png" title="CMIP5 multi-model mean precipitation, multi-model mean bias, multi-model mean of absolute error, multi-model mean of relative error; resembling Flato et al. (2013), Fig. 9.4." /></a></td>
Expand Down
38 changes: 16 additions & 22 deletions docs/model_evaluation/model_evaluation_on_gadi/model_evaluation_on_gadi_ilamb.md
View file
Original file line number Diff line number Diff line change
@@ -1,31 +1,29 @@
# ILAMB and IOMB on Gadi
# ILAMB on Gadi

## What are ILAMB and IOMB?
## What is ILAMB?

`ilamb` is a Python framework for both the <a href="https://www.ilamb.org/" target="_blank">International Land Model Benchmarking (ILAMB)</a> project and the <a href="https://cmec.llnl.gov/iomb.html" target="_blank">International Ocean Model Benchmark (IOMB)</a> package.
<br>
In brief, `ilamb` can be used to quantitatively compare a defined set of observable variables with a number of land and ocean models.
The <a href="https://www.ilamb.org/" target="_blank">International Land Model Benchmarking (<i>ILAMB</i>)</a> benchmarking system is a python framework used to quantitatively compare a defined set of observable variables with a number of land models.

???+ warning "Support Level: supported on <i>Gadi</i>, but not owned by ACCESS-NRI"
<!-- Who develped the tool? -->
ILAMB/IOMB is a community-developed climate model diagnostics and evaluation software package.
<i>ILAMB</i> is a community-developed climate model diagnostics and evaluation software package.
<!-- Code ownership and support -->
While ACCESS-NRI does not own the code, it actively supports the use of the software package on <i>Gadi</i>.
While ACCESS-NRI does not own the code, it actively supports the use of <i>ILAMB</i> software on <i>Gadi</i>.
<br>
<!--ILAMB development is primarily performed by the <a href="https://www.bgc-feedbacks.org/" target="_blank">RUBISCO</a> Science Focus Area and supported by the <a href="https://climatemodeling.science.energy.gov/program-area/regional-global-model-analysis" target="_blank">RGMA</a> Activity of the <a href="https://science.osti.gov/ber/Research/eessd" target="_blank">EESSD</a> division of the <a href="https://science.osti.gov/ber" target="_blank">BER</a> program in the United States Department of Energy’s Office of Science. -->
ACCESS-NRI provides access to the latest version of ILAMB/IOMB via the `xp65` <i>access-med conda environment</i> deployed on <i>Gadi</i>.
ACCESS-NRI provides access to the latest version of <i>ILAMB</i> via the `xp65` <i>access-med conda environment</i> deployed on <i>Gadi</i>.

This documentation is tailored to using `ilamb` on <i>Gadi</i> and, thus, it designed to supplement rather than replace the official documentation. Users are encouraged to read the <a href="https://www.ilamb.org/doc/" target="_blank">ILAMB documentation</a> and relevant <a href="https://www.ilamb.org/doc/tutorial.html" target="_blank">tutorials</a>.
This documentation is tailored to using <i>ILAMB</i> on <i>Gadi</i> and, hence, it supplements rather than replaces the official documentation. Users are encouraged to read the <a href="https://www.ilamb.org/doc/" target="_blank">ILAMB documentation</a> and relevant <a href="https://www.ilamb.org/doc/tutorial.html" target="_blank">tutorials</a>.

## Using ILAMB and IOMB on Gadi
## Using ILAMB on Gadi

`ilamb` is provided through both the `xp65` and `hh5` NCI projects on <i>Gadi</i>. Hence, you need to have an NCI account and be a member of these projects to use `ilamb`.
<i>ILAMB</i> is provided through both the `xp65` and `hh5` NCI projects on <i>Gadi</i>, so you need to have an NCI account and be a member of these projects to use it.

<div class="note">
To obtain an NCI account and join NCI projects, refer to <a href="\getting_started" target="_blank">Getting Started</a>.
</div>

For ACCESS-NRI documentation on how to run ILAMB and IOMB on <i>Gadi</i>, please visit:
For ACCESS-NRI documentation on how to run <i>ILAMB</i> on <i>Gadi</i>, visit:

<div class="card-container">
<a href="https://ilamb-workflow.readthedocs.io/en/latest/?badge=latest" class="vertical-card aspect-ratio2to1" target="_blank">
Expand All @@ -37,7 +35,7 @@ For ACCESS-NRI documentation on how to run ILAMB and IOMB on <i>Gadi</i>, please
</div>

<br>
To run `ilamb`, you need to execute the command `ilamb-run` with a number of arguments/ files:
To run <i>ILAMB</i>, you need to execute the command `ilamb-run` with a number of arguments/ files:
```
ilamb-run --config config.cfg --model_setup model_setup.txt --regions regions.txt
```
Expand All @@ -50,13 +48,13 @@ ilamb-run --config config.cfg --model_setup model_setup.txt --regions regions.tx
<code>regions.txt</code> defines the regions to be compared, e.g., <code>global</code>, <code>aust</code> (Australia), etc.
</ul>

While you can define these files yourself, ACCESS-NRI provides the necessary files and tools to set your model paths for running computations on <i>Gadi</i>. All you need to do is decide which observations and models you would like to compare.
While these files can be self-defined, ACCESS-NRI provides the necessary files and tools to set your model paths to run on <i>Gadi</i>. All you need to do is decide which observations and models you wish to compare.
<br>
NCI hosts replicas of the ILAMB and IOMB observational data sets through the NCI project `ct11` as well as a large amount of model outputs are available on <i>Gadi</i>, such as ACCESS model output.
NCI hosts replicas of the <i>ILAMB</i> observational data sets through the NCI project `ct11` as well as a large amount of model outputs are available on <i>Gadi</i>, such as ACCESS model output.

For more information, refer to <a href="\model_evaluation/model_evaluation_observational_catalogs" target="_blank">Observational Data Catalogue</a> on how to find observational data on NCI and <a href="\model_evaluation/model_evaluation_model_catalogs" target="_blank">ACCESS-NRI Intake Catalog</a> for how to find model outputs.
<br>
To learn more about how to adjust the <code>ilamb</code> setup, refer to the official <a href="https://www.ilamb.org/doc/" target="_blank">ILAMB documentation</a> and relevant <a href="https://www.ilamb.org/doc/tutorial.html" target="_blank">tutorials</a>.
To learn more about how to adjust the <i>ILAMB</i> setup, refer to the official <a href="https://www.ilamb.org/doc/" target="_blank">ILAMB documentation</a> and relevant <a href="https://www.ilamb.org/doc/tutorial.html" target="_blank">tutorials</a>.

## Showcase: CMIP6 comparisons and ACCESS ESM1.5 benchmarking

Expand All @@ -72,16 +70,12 @@ In the following example, the supported ACCESS Earth System Model (ESM) <a href=
- <a href="https://gmd.copernicus.org/articles/13/977/2020/" target="_blank">BCC ESM1 (Beijing Climate Center Earth System Model version 1)</a>
- <a href="https://gmd.copernicus.org/articles/12/4823/2019/gmd-12-4823-2019.html" target="_blank">CanESM5 (Canadian Earth System Model version 5)</a>

A large amount of benchmark comparisons have been defined in the configuration file.
<br>
The comparison of variables have been organised under different sections, such as the <i>Hydrology Cycle</i>.
Numerous benchmark comparisons have been defined in the configuration file. The comparison of variables have been organised under different sections, such as the <i>Hydrology Cycle</i>.
<br>
For other variables, such as the <i>gross primary productivity</i> `gpp`, one or more datasets are available. For example, the gross primary productivity measurements of <a href="https://fluxnet.org/data/fluxnet2015-dataset/" target="_blank">FLUXNET2015</a>.
<br>
<br>
By clicking on a row in the table, you can expand it to see the underlying datasets used.
<br>
In the below table, the colourmap extends from best values in purple to worse data in orange.
By clicking on a row in the table, you can expand it to see the underlying datasets used. The table's colourmap extends from best values in purple to worse data in orange.

<p align="center"><img align="center" width="50%" src="../../../assets/model_evaluation/ilamb_output_3.png" alt="Starting side of ILAMB output"></p>

Expand Down

0 comments on commit 7f19d17

Please sign in to comment.