ExaGO build failure on Frontier with PrgEnv-gnu

[balos1]

ExaGO fails when building xsdk with rocm and PrgEnv-gnu on Frontier:

[ 97%] Linking CXX executable opflow
cd /tmp/balos1/spack-stage/spack-stage-exago-1.6.0-lv7yzk2izzpxqifpvjibua2mxrhaptuj/spack-build-lv7yzk2/applications && /autofs/nccs-svm1_proj/csc326/balos1/frontier/spack-xsdk/opt/spack/linux-sles15-zen3/gcc-11.2.0/cmake-3.27.6-z7eq7nsiawlbjtf5feyxitpwragn33wf/bin/cmake -E cmake_link_script CMakeFiles/app_opflow.dir/link.txt --verbose=1
/opt/cray/pe/mpich/8.1.23/ofi/gnu/9.1/bin/mpicxx -O3 -DNDEBUG -fopenmp CMakeFiles/app_opflow.dir/opflow_main.cpp.o -o opflow  -Wl,-rpath,/autofs/nccs-svm1_proj/csc326/balos1/frontier/spack-xsdk/opt/spack/linux-sles15-zen3/gcc-11.2.0/umpire-6.0.0-vzfkkasbfj4rbwgl4ofspykgh35cdzsw/lib:/autofs/nccs-svm1_proj/csc326/balos1/frontier/spack-xsdk/opt/spack/linux-sles15-zen3/gcc-11.2.0/raja-0.14.0-goi63ekobgg7dahilgvn2tft6d3xlp6s/lib:/autofs/nccs-svm1_proj/csc326/balos1/frontier/spack-xsdk/opt/spack/linux-sles15-zen3/gcc-11.2.0/petsc-3.20.0-6ursdqegoopzhpqzgwro2o4rlioz5kfi/lib:/autofs/nccs-svm1_proj/csc326/balos1/frontier/spack-xsdk/opt/spack/linux-sles15-zen3/gcc-11.2.0/openblas-0.3.24-76epib4val2vdqe5f2ftpaw6qxxs4xyb/lib::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::: ../src/opflow/libexago_opflow.a /autofs/nccs-svm1_proj/csc326/balos1/frontier/spack-xsdk/opt/spack/linux-sles15-zen3/gcc-11.2.0/hiop-1.0.0-j3244ymuunmgwnwn53i7ghwaz5a7pfhe/lib/libhiop.a /autofs/nccs-svm1_proj/csc326/balos1/frontier/spack-xsdk/opt/spack/linux-sles15-zen3/gcc-11.2.0/umpire-6.0.0-vzfkkasbfj4rbwgl4ofspykgh35cdzsw/lib/libumpire.so /autofs/nccs-svm1_proj/csc326/balos1/frontier/spack-xsdk/opt/spack/linux-sles15-zen3/gcc-11.2.0/raja-0.14.0-goi63ekobgg7dahilgvn2tft6d3xlp6s/lib/libRAJA.so -ldl /opt/cray/pe/gcc/11.2.0/snos/lib64/libgomp.so /usr/lib64/libpthread.so ../src/pflow/libexago_pflow.a ../src/ps/libexago_ps.a ../src/utils/libexago_utils.a /autofs/nccs-svm1_proj/csc326/balos1/frontier/spack-xsdk/opt/spack/linux-sles15-zen3/gcc-11.2.0/petsc-3.20.0-6ursdqegoopzhpqzgwro2o4rlioz5kfi/lib/libpetsc.so /autofs/nccs-svm1_proj/csc326/balos1/frontier/spack-xsdk/opt/spack/linux-sles15-zen3/gcc-11.2.0/openblas-0.3.24-76epib4val2vdqe5f2ftpaw6qxxs4xyb/lib/libopenblas.so 
/usr/bin/ld: ../src/opflow/libexago_opflow.a(opflowregi.cpp.o): in function `OPFLOWModelRegisterAll(_p_OPFLOW*)':
opflowregi.cpp:(.text+0x2c8): undefined reference to `OPFLOWModelCreate_PBPOLRAJAHIOPSPARSE(_p_OPFLOW*)'
collect2: error: ld returned 1 exit status
make[2]: *** [applications/CMakeFiles/app_opflow.dir/build.make:111: applications/opflow] Error 1
make[2]: Leaving directory '/tmp/balos1/spack-stage/spack-stage-exago-1.6.0-lv7yzk2izzpxqifpvjibua2mxrhaptuj/spack-build-lv7yzk2'
make[1]: *** [CMakeFiles/Makefile2:555: applications/CMakeFiles/app_opflow.dir/all] Error 2
make[1]: *** Waiting for unfinished jobs....

Full log: spack-build-out.txt

[cameronrutherford]

@abhyshr and @wperkins this error is occuring with a build of exago+mpi+python+ipopt~rocm~cuda. Key here is ~ipopt means https://github.com/pnnl/ExaGO/blob/828db06af5e0345b641b6d03ef7e2456d13469ea/src/opflow/interface/opflowregi.cpp#L79 and https://github.com/pnnl/ExaGO/blob/828db06af5e0345b641b6d03ef7e2456d13469ea/src/opflow/model/power_bal_hiop/pbpolrajahiopsparse.cpp#L3 end up being guarded by different compiler macros.

@balos1 I am curious why you are building on Frontier without HIP, but thank you for catching this edge case.

[cameronrutherford]

We also don't print this particular bit of information at configure time (we should), but I assume hiop~sparse+raja was the hiop configuration built, and so minimal spack spec to repro on any platform is exago+hiop+raja~ipopt ^hiop+raja~sparse

[balos1]

@cameronrutherford I was not attempting to build without rocm. It looks like doing xsdk+rocm did not propagate the rocm variant to exago. Ill fix that.

[cameronrutherford]

pnnl/ExaGO#41 should close the original issue. Feel free to open new one with +rocm and close this IMO