Skip to content

Commit ea2f3cb

Browse files
nkoukpaizanCameron Rutherfordcameronrutherfordspack-auto-module
authored
Upgrade ascent build system and use [email protected] on CI platforms (#20)
* Boilerplate scripts to install modules on Ascent via submodule Spack * Fix '--nnodes'-->'-nodes' on Ascent * Improve Ascent env.sh * [email protected] on Ascent * Apply pre-commmit fixes * Relax constraints on exago dependencies on Ascent and build ~python * concretizer: reuse was causing several packages to be duplicated in the environment. Require clean concretizations on Ascent. * Minor module update on Ascent * Add LAPACK_LIBRARIES to Ascent base script. CMAKE was picking up python's openblas otherwise. * Error with unzip. * Apply pre-commmit fixes * Add working build on ascent. * Add working gcc11.2.0 spack spec. * Add Ascent Spack pipeline. [ascent-rebuild] * Update gcc version to 11.2.0 in base.sh [skip-ci] * Fix stages of Ascent pipeline [ascent-rebuild] * Add working ascent spack build. * Add hiop@develop force rebuild to PNNL CI [ascent-rebuild] [newell-rebuild] [deception-rebuild] [incline-rebuild]. * Update Ascent spack built tcl modules * Only test ascent on tcl module update [ci-skip] * Update base.sh to disable python on ascent [skip ci] * Remove LAPACK_LIBRARIES spec [ascent-test] * Update ascent.gitlab-ci.yml to fix needs/dependencies [ascent-test] * Update deception spack built tcl modules - [deception-test] * Try again with Python, but have Spack build it instead of using the external module [ascent-rebuild] * Force python rebuild on ascent and use [email protected] on incline [ascent-rebuild] [newell-rebuild] [incline-rebuild] * Pin [email protected] on all CI platforms [decetpion-rebuild] [ascent-rebuild] [newell-rebuild] [incline-rebuild] * Fix false positive/negative in Ascent pipelines [deception-rebuild] [ascent-test] * Update incline spack built tcl modules - [incline-test] * Update newell spack built tcl modules - [newell-test] * Fix HiOp spec on Ascent [ascent-rebuild]. * Update deception spack built tcl modules - [deception-test] * Update CPU Spack build with issue for each failing build [ci skip] * Update Ascent spack built tcl modules [ascent-test] * Add 1.0.0 dep into CHANGELOG. * Add ascent-skip to CI to get tests passing [ascent-test] --------- Co-authored-by: nkoukpaizan <[email protected]> Co-authored-by: Cameron Rutherford <[email protected]> Co-authored-by: cameronrutherford <[email protected]> Co-authored-by: spack-auto-module <[email protected]>
1 parent 828db06 commit ea2f3cb

28 files changed

+656
-413
lines changed

‎.github/workflows/spack_cpu_build.yaml

+2-2
Original file line numberDiff line numberDiff line change
@@ -12,8 +12,8 @@ jobs:
1212
strategy:
1313
matrix:
1414
spack_spec:
15-
# Minimal Build
16-
# See #18
15+
# Minimal Build(s)
16+
# See #39 - ~python~mpi causes issues
1717
# - exago@develop~mpi~ipopt~hiop~python~raja
1818
- exago@develop+mpi~ipopt+hiop~python~raja ^openmpi
1919
# See #18 - +hiop~mpi causes issues

‎.gitignore

+2
Original file line numberDiff line numberDiff line change
@@ -12,3 +12,5 @@ node_modules
1212
.env
1313
package-lock.json
1414
config.py
15+
spack_*
16+
hiop/

‎.gitlab/ornl/ascent.gitlab-ci.yml

+50-5
Original file line numberDiff line numberDiff line change
@@ -102,6 +102,7 @@
102102
- rm -rf "$WORKDIR"
103103

104104
Ascent Build:
105+
needs: []
105106
stage: build
106107
tags:
107108
- nobatch
@@ -112,27 +113,71 @@ Ascent Build:
112113

113114
Ascent Test:
114115
stage: test
115-
dependencies:
116-
- 'Ascent Build'
116+
needs: ['Ascent Build']
117117
tags:
118118
- batch
119119
variables:
120120
# Don't clone for test jobs
121121
GIT_STRATEGY: none
122-
CTESTARGS: "--output-on-failure"
122+
CTESTARGS: "--output-on-failure -LE ascent-skip "
123123
extends:
124124
- .ascent_rules
125125
- .ascent_test_script_template
126126
- .ascent_environment_template
127127

128+
# -- Spack Module Rebuilding
129+
130+
Ascent-Module-Init:
131+
needs: []
132+
extends: .ascent_environment_template
133+
variables:
134+
GIT_SUBMODULE_STRATEGY: recursive
135+
rules:
136+
- if: '$CI_COMMIT_TITLE =~ /\[ascent-rebuild\]/'
137+
when: always
138+
stage: build
139+
tags:
140+
- nobatch
141+
script:
142+
- set -xv
143+
- mkdir -p "$WORKDIR"
144+
- cp -r . "$WORKDIR"
145+
- cd "$WORKDIR"
146+
- MY_CLUSTER=ascent ./buildsystem/spack/binary_mirror.sh
147+
- res=$?
148+
- exit $res
149+
150+
Ascent-Module-Build:
151+
needs: ["Ascent-Module-Init"]
152+
rules:
153+
- if: '$CI_COMMIT_TITLE =~ /\[ascent-rebuild\]/'
154+
when: always
155+
extends: .ascent_environment_template
156+
stage: test
157+
tags:
158+
- batch
159+
script:
160+
- cd "$WORKDIR"
161+
- git checkout -b ${CI_COMMIT_REF_NAME}
162+
- ./buildsystem/spack/ascent/bsub.sh
163+
- git add ./buildsystem/spack/ascent/modules/*
164+
- git config user.name "spack-auto-module"
165+
- git config user.email "[email protected]"
166+
- git commit -m "Update Ascent spack built tcl modules [ascent-test]"
167+
- git remote set-url origin https://gitlab-ci-token:${SPACK_GIT_TOKEN}@github.com/pnnl/ExaGO.git
168+
- git pull --rebase origin ${CI_COMMIT_REF_NAME}
169+
- git push origin ${CI_COMMIT_REF_NAME}
170+
after_script:
171+
- cd "$WORKDIR/.."
172+
- rm -rf "$WORKDIR"
173+
128174
# -- Reporting Ascent Status to PNNL GitHub
129175
.report-job:
130-
when: always
131176
variables:
132177
MY_CLUSTER: Ascent
133178
GIT_STRATEGY: none
134179
STATUS_PROJECT: pnnl/ExaGO
135-
STATUS_NAME: ppc64le/gcc@10.2/cuda@11.4.2/v100@70
180+
STATUS_NAME: ppc64le/gcc@11.2/cuda@11.8/v100@70
136181
tags:
137182
- nobatch
138183
extends: .ascent_environment_template

‎CHANGELOG.md

+3
Original file line numberDiff line numberDiff line change
@@ -8,9 +8,12 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
88
## [develop]
99

1010
### General
11+
- Transition from GitLab to GitHub
12+
- ExaGO now depends on HiOp 1.0.0. Next release will require HiOp 1.0.1+
1113

1214
### Build system
1315
- Add CI on Incline with Spack module pipelines (AMD / HIP GPU machine at PNNL)
16+
- Add Spack module pipelines to Ascent
1417

1518
### PS
1619

‎buildsystem/gcc-cuda/ascent/base.sh

+17
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,17 @@
1+
#!/bin/bash
2+
3+
export MY_CLUSTER=ascent
4+
5+
module reset
6+
7+
# Load system modules
8+
module load gcc/11.2.0
9+
module load spectrum-mpi/10.4.0.3-20210112
10+
11+
export CC=$(which gcc)
12+
export CXX=$(which g++)
13+
export FC=$(which gfortran)
14+
15+
export MY_PETSC_DIR=$PETSC_DIR
16+
export EXTRA_CMAKE_ARGS="$EXTRA_CMAKE_ARGS -DEXAGO_CTEST_LAUNCH_COMMAND='jsrun -g 1 -n 1'"
17+
export EXTRA_CMAKE_ARGS="$EXTRA_CMAKE_ARGS -DEXAGO_ENABLE_PYTHON=OFF"

‎buildsystem/gcc-cuda/ascent/spack.yaml

-75
This file was deleted.

‎buildsystem/gcc-cuda/ascentVariables.sh

+6-122
Original file line numberDiff line numberDiff line change
@@ -1,125 +1,9 @@
1-
export MY_CLUSTER=ascent
1+
#!/bin/bash
22

3-
module purge
3+
export SRCDIR=${SRCDIR:-$PWD}
44

5-
module use -a /gpfs/wolf/proj-shared/csc359/src/spack/share/spack/modules/linux-rhel8-power9le
5+
# Shared system configuration
6+
source $SRCDIR/buildsystem/gcc-cuda/ascent/base.sh
67

7-
# Load spack modules
8-
# [email protected]%[email protected] arch=linux-rhel8-power9le
9-
module load exasgd-blt/0.4.1/gcc-10.2.0-ndtzigt
10-
# [email protected]%[email protected]+cuda~ipo~rocm~tests build_type=RelWithDebInfo cuda_arch=70 arch=linux-rhel8-power9le
11-
module load exasgd-camp/0.2.2/cuda-11.4.2/gcc-10.2.0-2xxjut5
12-
# [email protected]%[email protected]~doc+ncurses+ownlibs~qt build_type=Release arch=linux-rhel8-power9le
13-
module load exasgd-cmake/3.22.2/gcc-10.2.0-sj55k3d
14-
# [email protected]%[email protected]+blas arch=linux-rhel8-power9le
15-
module load exasgd-coinhsl/2015.06.23/gcc-10.2.0-zr37tzv
16-
# [email protected]%[email protected] arch=linux-rhel8-power9le
17-
module load exasgd-cub/1.16.0/gcc-10.2.0-zeukz4z
18-
# [email protected]%[email protected]~allow-unsupported-compilers~dev arch=linux-rhel8-power9le
19-
module load exasgd-cuda/11.4.2/gcc-10.2.0-yf564wn
20-
# [email protected]%[email protected] arch=linux-rhel8-power9le
21-
module load exasgd-diffutils/3.8/gcc-10.2.0-n2gttub
22-
# [email protected]_experimental%[email protected]+cuda~develtools~full_optimizations~hwloc~ipo~oneapi+openmp~rocm+shared build_system=cmake build_type=Debug cuda_arch=70 dev_path=/gpfs/wolf/proj-shared/csc359/src/ginkgo arch=linux-rhel8-power9le
23-
module load exasgd-ginkgo/1.5.0.glu_experimental/cuda-11.4.2/gcc-10.2.0-h4v5snw
24-
# [email protected]%[email protected] libs=shared,static arch=linux-rhel8-power9le
25-
module load exasgd-gmp/6.2.1/gcc-10.2.0-6oxkdwz
26-
# gnuconfig@2021-08-14%[email protected] arch=linux-rhel8-power9le
27-
module load exasgd-gnuconfig/2021-08-14/gcc-10.2.0-j2pelq3
28-
# [email protected]%[email protected]~cxx~fortran~hl~ipo~java+mpi+shared~szip~threadsafe+tools api=default build_type=RelWithDebInfo arch=linux-rhel8-power9le
29-
module load exasgd-hdf5/1.12.2/spectrum-mpi-10.4.0.3-20210112/gcc-10.2.0-yh642ea
30-
# hiop@develop%[email protected]+cuda+cusolver+deepchecking+ginkgo~ipo~jsrun+kron+mpi+raja~rocm~shared+sparse build_system=cmake build_type=RelWithDebInfo cuda_arch=70 dev_path=/gpfs/wolf/proj-shared/csc359/src/nicholson-hiop arch=linux-rhel8-power9le
31-
module load exasgd-hiop/develop/cuda-11.4.2/spectrum-mpi-10.4.0.3-20210112/gcc-10.2.0-ucuclxs
32-
# [email protected]%[email protected]~complex~cuda~debug+fortran~gptune~int64~internal-superlu~mixedint+mpi~openmp~rocm+shared~superlu-dist~unified-memory arch=linux-rhel8-power9le
33-
module load exasgd-hypre/2.24.0/spectrum-mpi-10.4.0.3-20210112/gcc-10.2.0-oybqs3s
34-
# [email protected]%[email protected]+coinhsl~debug~metis~mumps build_system=autotools arch=linux-rhel8-power9le
35-
module load exasgd-ipopt/3.12.10/gcc-10.2.0-i4gqiqg
36-
# [email protected]%[email protected] libs=shared,static arch=linux-rhel8-power9le
37-
module load exasgd-libiconv/1.16/gcc-10.2.0-cpn6euq
38-
# [email protected]%[email protected]+cuda+fortran~ipo~rocm+shared build_type=RelWithDebInfo cuda_arch=70 arch=linux-rhel8-power9le
39-
module load exasgd-magma/2.6.2/cuda-11.4.2/gcc-10.2.0-ffujy7o
40-
# [email protected]%[email protected]~gdb~int64~real64+shared build_type=Release patches=4991da9,b1225da arch=linux-rhel8-power9le
41-
module load exasgd-metis/5.1.0/gcc-10.2.0-kdt6x4i
42-
# [email protected]%[email protected] libs=shared,static arch=linux-rhel8-power9le
43-
module load exasgd-mpfr/4.1.0/gcc-10.2.0-ua5zak3
44-
# [email protected]%[email protected]~bignuma~consistent_fpcsr~ilp64+locking+pic+shared symbol_suffix=none threads=none arch=linux-rhel8-power9le
45-
module load exasgd-openblas/0.3.20/gcc-10.2.0-lkjyfws
46-
# [email protected]%[email protected]~gdb~int64~ipo+shared build_type=RelWithDebInfo patches=4f89253,50ed208,704b84f arch=linux-rhel8-power9le
47-
module load exasgd-parmetis/4.0.3/spectrum-mpi-10.4.0.3-20210112/gcc-10.2.0-gqz32yk
48-
# [email protected]%[email protected]~X~batch~cgns~complex~cuda~debug+double~exodusii~fftw+fortran~giflib+hdf5~hpddm~hwloc+hypre~int64~jpeg~knl~kokkos~libpng~libyaml~memkind+metis~mkl-pardiso~mmg~moab~mpfr+mpi~mumps~openmp~p4est~parmmg~ptscotch~random123~rocm~saws~scalapack+shared~strumpack~suite-sparse+superlu-dist~tetgen~trilinos~valgrind build_system=generic clanguage=C arch=linux-rhel8-power9le
49-
module load exasgd-petsc/3.18.1/spectrum-mpi-10.4.0.3-20210112/gcc-10.2.0-cwok3k4
50-
# [email protected]%[email protected] arch=linux-rhel8-power9le
51-
module load exasgd-pkgconf/1.8.0/gcc-10.2.0-nngfigd
52-
# [email protected]%[email protected] arch=linux-rhel8-power9le
53-
module load exasgd-py-attrs/21.4.0/gcc-10.2.0-av2tzhn
54-
# [email protected]%[email protected] arch=linux-rhel8-power9le
55-
module load exasgd-py-flit-core/3.6.0/gcc-10.2.0-wdoqgro
56-
# [email protected]%[email protected] arch=linux-rhel8-power9le
57-
module load exasgd-py-importlib-metadata/4.8.2/gcc-10.2.0-klg3xix
58-
# [email protected]%[email protected] arch=linux-rhel8-power9le
59-
module load exasgd-py-iniconfig/1.1.1/gcc-10.2.0-o6bmfak
60-
# [email protected]%[email protected] arch=linux-rhel8-power9le
61-
module load exasgd-py-mpi4py/3.1.2/spectrum-mpi-10.4.0.3-20210112/gcc-10.2.0-q53tcvd
62-
# [email protected]%[email protected] arch=linux-rhel8-power9le
63-
module load exasgd-py-packaging/21.3/gcc-10.2.0-ej3e6zm
64-
# [email protected]%[email protected] arch=linux-rhel8-power9le
65-
module load exasgd-py-pip/21.3.1/gcc-10.2.0-awkoczv
66-
# [email protected]%[email protected] arch=linux-rhel8-power9le
67-
module load exasgd-py-pluggy/1.0.0/gcc-10.2.0-tlqdjwe
68-
# [email protected]%[email protected] arch=linux-rhel8-power9le
69-
module load exasgd-py-py/1.11.0/gcc-10.2.0-fwbtxbd
70-
# [email protected]%[email protected] arch=linux-rhel8-power9le
71-
module load exasgd-py-pyparsing/3.0.6/gcc-10.2.0-qxfb6l5
72-
# [email protected]%[email protected] arch=linux-rhel8-power9le
73-
module load exasgd-py-pytest/6.2.5/gcc-10.2.0-2vv5hdj
74-
# [email protected]%[email protected] arch=linux-rhel8-power9le
75-
module load exasgd-py-setuptools/59.4.0/gcc-10.2.0-elx3acw
76-
# [email protected]%[email protected]+toml arch=linux-rhel8-power9le
77-
module load exasgd-py-setuptools-scm/6.3.2/gcc-10.2.0-xdvaqbm
78-
# [email protected]%[email protected] arch=linux-rhel8-power9le
79-
module load exasgd-py-toml/0.10.2/gcc-10.2.0-6lhs4ix
80-
# [email protected]%[email protected] arch=linux-rhel8-power9le
81-
module load exasgd-py-tomli/1.2.2/gcc-10.2.0-4glf4eg
82-
# [email protected]%[email protected] arch=linux-rhel8-power9le
83-
module load exasgd-py-typing-extensions/4.1.1/gcc-10.2.0-qg6k6g2
84-
# [email protected]%[email protected] arch=linux-rhel8-power9le
85-
module load exasgd-py-wheel/0.37.0/gcc-10.2.0-tg3twdl
86-
# [email protected]%[email protected] arch=linux-rhel8-power9le
87-
module load exasgd-py-zipp/3.6.0/gcc-10.2.0-4vxciug
88-
# [email protected]%[email protected]+bz2+ctypes+dbm~debug+ensurepip+libxml2+lzma~nis~optimizations+pic+pyexpat+pythoncmd+readline+shared+sqlite3+ssl~tix~tkinter~ucs4+uuid+zlib patches=c129e34 arch=linux-rhel8-power9le
89-
module load exasgd-python/3.6.8/gcc-10.2.0-273robh
90-
# [email protected]%[email protected]+cuda~examples~exercises~ipo+openmp~rocm+shared~tests build_type=RelWithDebInfo cuda_arch=70 arch=linux-rhel8-power9le
91-
module load exasgd-raja/0.14.0/cuda-11.4.2/gcc-10.2.0-hkkig4v
92-
# [email protected]%[email protected] arch=linux-rhel8-power9le
93-
module load exasgd-spectrum-mpi/10.4.0.3-20210112/gcc-10.2.0-c5uqeqr
94-
# [email protected]%[email protected]~cuda~graphblas~openmp+pic~tbb arch=linux-rhel8-power9le
95-
module load exasgd-suite-sparse/5.10.1/gcc-10.2.0-sq5isrj
96-
# [email protected]%[email protected]~cuda~int64~ipo~openmp~rocm+shared build_type=RelWithDebInfo patches=8da9e21 arch=linux-rhel8-power9le
97-
module load exasgd-superlu-dist/7.2.0/spectrum-mpi-10.4.0.3-20210112/gcc-10.2.0-7cwmhkg
98-
# [email protected]%[email protected]~c+cuda~device_alloc~deviceconst~examples~fortran~ipo~numa~openmp~rocm~shared build_type=RelWithDebInfo cuda_arch=70 tests=none arch=linux-rhel8-power9le
99-
module load exasgd-umpire/6.0.0/cuda-11.4.2/gcc-10.2.0-selkipw
100-
# [email protected]%[email protected]+optimize+pic+shared patches=0d38234 arch=linux-rhel8-power9le
101-
module load exasgd-zlib/1.2.12/gcc-10.2.0-lfmc24z
102-
103-
# Load system modules
104-
module load cuda/11.4.2
105-
module load gcc/10.2.0
106-
module load spectrum-mpi/10.4.0.3-20210112
107-
module load cmake/3.22.2
108-
109-
export MY_PETSC_DIR=$PETSC_DIR
110-
export EXTRA_CMAKE_ARGS="$EXTRA_CMAKE_ARGS -DEXAGO_CTEST_LAUNCH_COMMAND='jsrun -g 1 -n 1'"
111-
112-
[ -f $PWD/nvblas.conf ] && rm $PWD/nvblas.conf
113-
cat > $PWD/nvblas.conf <<-EOD
114-
NVBLAS_LOGFILE nvblas.log
115-
NVBLAS_CPU_BLAS_LIB $OPENBLAS_LIBRARY_DIR/libopenblas.so
116-
NVBLAS_GPU_LIST ALL
117-
NVBLAS_TILE_DIM 2048
118-
NVBLAS_AUTOPIN_MEM_ENABLED
119-
EOD
120-
export NVBLAS_CONFIG_FILE=$PWD/nvblas.conf
121-
echo "Generated $PWD/nvblas.conf"
122-
123-
export CC=$(which gcc)
124-
export CXX=$(which g++)
125-
export FC=$(which gfortran)
8+
# Spack modules
9+
source $SRCDIR/buildsystem/spack/$MY_CLUSTER/modules/dependencies.sh

0 commit comments

Comments
 (0)