Create page: Set up Spack for building ACCESS models on Gadi

[atteggiani]

Fixes #785.

• Updated classes and cards for aditions to getting_started
• Created 'Set up Spack for model builds' page inside 'Getting Started'

[github-actions]

PR preview
⚠️ There was an error in the pr-preview deployment. For more information, please check the Actions tab.
2024-09-19 10:39 AEST

[harshula]

Version 1.3 of my preferred text and formatting is:

Set up Spack for building ACCESS models

Spack is a build-from-source package manager, specifically designed to simplify the installation of scientific software on supercomputers.

To use Spack, please familiarise yourself with the Basic Usage instructions and Environments.

We also recommend that you refer to the Spack 101 Tutorial.

Prerequisites

These instructions are tailored specifically for Gadi. To use Spack on Gadi, you must have an NCI account. For instructions on how to set up an account, refer to Set Up your NCI Account.

The steps for setting up Spack on Gadi are:

Create a directory on the filesystem where Spack will be installed. e.g. /g/data/$PROJECT/$USER/<myspack>. This directory shall be referred to as <myspack>.

mkdir <myspack>
cd <myspack>

Clone the relevant git repositories:

git clone -c feature.manyFiles=true https://github.com/ACCESS-NRI/spack.git --branch releases/v0.22 --single-branch --depth=1
git clone https://github.com/ACCESS-NRI/spack-packages.git --branch main
git clone https://github.com/ACCESS-NRI/spack-config.git --branch main

Link spack-config configurations files to the Spack instance:

ln -s -r -v spack-config/v0.22/gadi/* spack/etc/spack/

Test Spack (OPTIONAL)

To test Spack we will create an ACCESS-OM2 environment and build the relevant packages. Then, we will uninstall all the packages and remove the environment.

module purge
cd <myspack>
. spack-config/spack-enable.bash
git clone https://github.com/ACCESS-NRI/ACCESS-OM2.git
spack env create access-om2 ACCESS-OM2/spack.yaml
spack env activate access-om2
spack find
spack concretize -f
spack install --verbose
spack find
spack uninstall --remove --all
spack env deactivate
spack env rm access-om2
rm -rf ACCESS-OM2

[atteggiani]

To use Spack, please familiarise yourself with the Basic Usage instructions and Environments.

Written like this it seems as if, to use Spack for building a model, a user needs to read those pages (basic usage and environments). However, reading those pages is a step that might be helpful to get more comfortable with Spack (and maybe do some more advanced stuff).

The minimum steps for building a new version of an ACCESS model (for example a version with the source code of a component changed) should be as simple as running a few commands, and should not require the user to read extra documentation about the usage of Spack.

Create a directory on the filesystem where Spack will be installed. This directory shall be refered to as myspack. e.g. /g/data/$PROJECT/$USER/myspack

Creating a directory for the Spack installation is not a prerequisite (in fact is also optional, although reccommended), but rather an actual (optional) step for the setup.
For the example name we can use whatever you prefer (if you want to use myspack that's fine).

General comments:

1. Having the steps as subparagraphs (instead of numbered items) is consistent with the rest of the Hive Docs and makes it easier to link to and have the table of contents on the right-side of the website.
2. Is it possible to have the test building something smaller/quicker? Building ACCESS-OM2 takes a long time, and it might not be the best for a quick test.

[harshula]

Hi @atteggiani , I think it's important to appreciate that these instructions are for advanced users that plan to modify the source code and/or change dependencies of ACCESS models. They are not for users that just want to run unmodified ACCESS models.

To use Spack, please familiarise yourself with the Basic Usage instructions and Environments.

Written like this it seems as if, to use Spack for building a model, a user needs to read those pages (basic usage and environments). However, reading those pages is a step that might be helpful to get more comfortable with Spack (and maybe do some more advanced stuff).

We want the user to read the "basic usage" and "environments" sections, if they are going to use Spack to build ACCESS models.

Create a directory on the filesystem where Spack will be installed. This directory shall be refered to as myspack. e.g. /g/data/$PROJECT/$USER/myspack

Creating a directory for the Spack installation is not a prerequisite (in fact is also optional, although reccommended), but rather an actual (optional) step for the setup. For the example name we can use whatever you prefer (if you want to use myspack that's fine).

It is a prerequisite. We want the user to keep all the Spack related files/directories in a new directory. I'll change myspack to <myspack> to make it clearer that it is an arbitrary directory name.

General comments:

1. Having the steps as subparagraphs (instead of numbered items) is consistent with the rest of the Hive Docs and makes it easier to link to and have the table of contents on the right-side of the website.

If you prefer to remove the enumeration, remove the numbers but please keep the prose.

2. Is it possible to have the test building something smaller/quicker? Building ACCESS-OM2 takes a long time, and it might not be the best for a quick test.

Unfortunately, it's the quickest ACCESS model to build. You make a good point, perhaps we need to create a Spack environment just for testing purposes. That can be an objective of the second iteration of these instructions.

[atteggiani]

Hi @harshula, thank you for your comments.

I think it's important to appreciate that these instructions are for advanced users that plan to modify the source code and/or change dependencies of ACCESS models. They are not for users that just want to run unmodified ACCESS models.

I would not consider users that plan to modify the source code and change dependencies that "advanced", in the sense that they might be researchers that don't have a lot of experience with ACCESS models and Gadi.
Most of the users that start running an "unmodified" ACCESS model, will then want to run a "modified" version of it. There is almost no point in running a model as it is only to produce output data, because the output data is already available elsewhere.
It is true that not all modifications require a rebuild, but even a "minor" source code change of a component requires a rebuild.
This means the build process should be understandable and approachable (pretty easily) by anyone who already knows how to run an "unmodified" model.

We want the user to read the "basic usage" and "environments" sections, if they are going to use Spack to build ACCESS models.

Spack is an amazing tool for simplifying builds and making them more robust (from a reproducibility and scientific point of view).
However, it is not the main objective and interest of the user. A user's main objective is to effectively build an ACCESS model. If possible, without having to deal with too many other pieces of softwares and steps in the process.
The process of building an ACCESS model using Spack shouldn't become more onerous (for the user) than building a model without Spack.

I think we should have a Spack workflow that minimizes the knowledge a user needs to have about Spack to be able to build a specific ACCESS model (and I think it is possible despite current Spack/modules limitations).
For this reason, reading Spack's "basic usage" and "environments" documentation is definitely recommended but should not be seen as a requirement.

It is a prerequisite. We want the user to keep all the Spack related files/directories in a new directory.

In the Hive Docs we list as prerequisites all the steps that would prevent users from doing further steps listed in the guide and, for this reason, need to be taken care of before following the actual guide.
In this case, I can start with the cloning of the repos and proceed with the other steps without issues. I would still have a working instance of Spack (probably in my $HOME directory on Gadi). It is not recommended for different reasons, but if anyone wants to do it this way, it works.

If you prefer to remove the enumeration, remove the numbers but please keep the prose.

The prose is similar. If you have suggestions bout that please comment on the specific portion and I will address the comments/suggestions.

Unfortunately, it's the quickest ACCESS model to build. You make a good point, perhaps we need to create a Spack environment just for testing purposes. That can be an objective of the second iteration of these instructions.

Ok, for the time being is fine. But I agree it would be best to have a Spack environment just for testing.

Thank you!

[heidinett]

Warning message:
This page is intended for experienced users and collaborators developing ACCESS model configurations

[harshula]

Hi @atteggiani , Thanks for the chat, I've updated the text and formatting to v1.2.
Hi @heidinett , re: "... developing ACCESS model configurations" , Can we please avoid mentioning "configurations"?

[heidinett]

Hi @atteggiani , Thanks for the chat, I've updated the text and formatting to v1.2. Hi @heidinett , re: "... developing ACCESS model configurations" , Can we please avoid mentioning "configurations"?

Sure! Happy to drop "configurations" It was just a draft to get an idea down that @atteggiani will finesse :)

[harshula]

Thanks @heidinett ! Hi @atteggiani , I've started a Test build on Gadi to reproduce the build failure.

[harshula]

Hi @atteggiani , The test build on Gadi succeeded. What was the error message you received?

[atteggiani]

Warning message: This page is intended for experienced users and collaborators developing ACCESS model configurations

I propose "This page is tailored to experienced users and collaborators developing ACCESS models."
@harshula @heidinett

[atteggiani]

Hi @atteggiani , The test build on Gadi succeeded. What was the error message you received?

Mine failed again.
The log file is here:
/scratch/tm70/dm5220/tmp/spack-stage/spack-stage-oasis3-mct-git.2023.11.09_2023.11.09-d6o4vzy5pbqhv5ypzrudo25ixgc7vjec/spack-build-out.txt

[atteggiani]

Hi @atteggiani , The test build on Gadi succeeded. What was the error message you received?

Mine failed again. The log file is here: /scratch/tm70/dm5220/tmp/spack-stage/spack-stage-oasis3-mct-git.2023.11.09_2023.11.09-d6o4vzy5pbqhv5ypzrudo25ixgc7vjec/spack-build-out.txt

The main error seems to be the following:

/scratch/tm70/dm5220/tmp/spack-stage/spack-stage-oasis3-mct-git.2023.11.09_2023.11.09-d6o4vzy5pbqhv5ypzrudo25ixgc7vjec/spack-src/lib/psmile/src/mod_oasis_part.F90(8): error #7002: Error in opening the compiled module file.  Check INCLUDE paths.   [MCT_MOD]

[harshula]

Actual error was:

ar libmpeu.a get_zeits.o m_IndexBin_char.o m_IndexBin_integer.o m_IndexBin_logical.o m_List.o m_MergeSorts.o m_Filename.o m_ErrorHandler.o m_FcComms.o m_Permuter.o m_SortingTools.o m_String.o m_StrTemplate.o m_chars.o m_die.o m_dropdead.o m_FileResolv.o m_flow.o m_inpak90.o m_ioutil.o m_mall.o m_mpif.o m_mpif90.o m_mpout.o m_rankMerge.o m_realkinds.o m_stdio.o m_TraceBack.o m_zeit.o
ar: two different operation options specified

[harshula]

I've updated the text to v1.3. I added module purge to the test instructions. Davide had done the test with conda environments loaded.

[atteggiani]

The build was failing because I had theconda/analysis3 module loaded.
After removing the module, the build succeeds!

[atteggiani]

I found out what the error is caused by.

It is the AR environment variable (and potentially also GCC_AR) set, for example, by the conda/analysis3 module).

I tried purging the environment, then exporting the AR variable to be the same ar executable that conda/analysis3 would export (/g/data/hh5/public/apps/miniconda3/envs/analysis3-24.04/bin/x86_64-conda-linux-gnu-ar) and the build still fails.

After I unset the AR variable the build succeeds.

[KAUR1984]

Thanks for nice and compact PR @atteggiani ;)! Looks good to me.

[harshula]

I only reviewed spack.md. Approved.