cmake: move kokkos files into subdirectory

[rbberger]

This refactor moves the kokkos interop code into its own subdirectory. This allows us to isolate, for the most part, the Kokkos compilation from the rest of the Realm build. This removes the client restriction to use Kokkos with
-DKokkos_ENABLE_COMPILE_AS_CMAKE_LANGUAGE=ON.

We end up with two cases:

Shared build

realm_kokkos is a shared library and self-contained. Linking to it doesn't propagate link options.

Static build

realm_kokkos is a static library and inherits the Kokkos INTERFACE link options. This forces us (and our clients) to use the same linker/wrapper as Kokkos.

Spack package changes

The new Realm spack package enforces these two cases and acts accordingly. The most isolated build is the shared library. For the static build we must use the same compiler/wrapper as Kokkos. In addition, we must ensure that Realm uses the same GCC and Clang as Kokkos.

spack/spack-packages#2941

Below is a Spack environment used for verifying the builds in various combinations:

Note

Not all combinations in the Spack test environment pass due to Kokkos and CMake issues in certain combinations, but this change significantly increases the functional combinations. (40/44)
Known Issues:

• kokkos+shared+cuda+cmake_lang %cxx=clang
• kokkos~shared+cuda_cmake_lang %cxx=clang

The problem here is that Kokkos uses clang++ as CMAKE_CUDA_COMPILER, however, there is no good way of changing that compiler in a subdirectory. Mixing CMAKE_CUDA_COMPILER=nvcc and CMAKE_CUDA_COMPILER=clang++ in a single CMake build seems to cause issues. So the better solution will be to use clang all the way in that case. However, as we've seen, this requires: #368

Spack Test Environment for testing combinations (on Ubuntu 24.04 with CUDA 12.8 and ROCm 6.4.3 installed)

spack:
  # add package specs to the `specs` list
  specs:
  - matrix:
    - [realm]
    - [~openmp, +openmp]
    - [+cuda cuda_arch=86 +cuda_unsupported_compiler]
    - [+shared, ~shared]
    - ['%cxx=gcc', '%cxx=clang']
    - [~kokkos, +kokkos ^kokkos+wrapper~cmake_lang, +kokkos ^kokkos+cmake_lang~wrapper]
  - matrix:
    - [realm]
    - [~openmp, +openmp]
    - [+cuda cuda_arch=86 +cuda_unsupported_compiler]
    - [+shared, ~shared]
    - ['%cxx=clang']
    - [+kokkos ^kokkos~wrapper~cmake_lang]
  - matrix:
    - [realm]
    - [~openmp, +openmp]
    - [+rocm amdgpu_target=gfx942]
    - [+shared, ~shared]
    - [~kokkos, +kokkos]
    - ['%cxx=gcc', '%cxx=clang']
  view: false
  concretizer:
    unify: false
  packages:
    realm:
      require:
        - "@git.kokkos_cmake_refactor=main"
      package_attributes:
        git: https://github.com/rbberger/realm.git
    llvm:
      require:
        - "@19"
      externals:
      - spec: [email protected]+clang+flang~lld~lldb
        prefix: /usr
        extra_attributes:
          compilers:
            c: /usr/bin/clang-19
            cxx: /usr/bin/clang++-19
            fortran: /usr/bin/flang-19
    llvm-amdgpu:
      externals:
      - spec: [email protected]
        prefix: /opt/rocm-6.4.3/lib/llvm
      buildable: false
    gcc:
      require:
        - "@13"
      externals:
      - spec: [email protected] languages:='c,c++,fortran'
        prefix: /usr
        extra_attributes:
          compilers:
            c: /usr/bin/gcc
            cxx: /usr/bin/g++
            fortran: /usr/bin/gfortran
    kokkos:
      prefer: ['@4.7.02']
    kokkos-nvcc-wrapper:
      prefer: ['@4.7.02']
    cmake:
      externals:
      - spec: [email protected]
        prefix: /usr
      buildable: false
    cuda:
      require:
      - '@12.8 +allow-unsupported-compilers'
      externals:
      - spec: [email protected] +allow-unsupported-compilers
        prefix: /usr/local/cuda-12.8
      buildable: false
    hip:
      version: [6.4.3]
      externals:
      - spec: [email protected]
        prefix: /opt/rocm-6.4.3
      buildable: false
    hsa-rocr-dev:
      externals:
      - spec: [email protected]
        prefix: /opt/rocm-6.4.3
      buildable: false
    rocsolver:
      externals:
      - spec: [email protected]
        prefix: /opt/rocm-6.4.3
      buildable: false
    rocblas:
      externals:
      - spec: [email protected]
        prefix: /opt/rocm-6.4.3
      buildable: false
    rocthrust:
      externals:
      - spec: [email protected]
        prefix: /opt/rocm-6.4.3
      buildable: false
    rocprim:
      externals:
      - spec: [email protected]
        prefix: /opt/rocm-6.4.3
      buildable: false
    rocsparse:
      externals:
      - spec: [email protected]
        prefix: /opt/rocm-6.4.3
      buildable: false
    rocrand:
      externals:
      - spec: [email protected]
        prefix: /opt/rocm-6.4.3
      buildable: false
    rocm-smi-lib:
      externals:
      - spec: [email protected]
        prefix: /opt/rocm-6.4.3
      buildable: false
    rocprofiler-dev:
      externals:
      - spec: [email protected]
        prefix: /opt/rocm-6.4.3
      buildable: false
    roctracer-dev:
      externals:
      - spec: [email protected]
        prefix: /opt/rocm-6.4.3
      buildable: false

[rbberger]

@elliottslaughter @muraj This is ready for review and testing on your end. Together with the Spack package change, this should significantly improve the situation.

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 27.74%. Comparing base (0ef7edc) to head (1510369).
⚠️ Report is 1 commits behind head on main.
✅ All tests successful. No failed tests found.

Additional details and impacted files

@@           Coverage Diff           @@
##             main     #385   +/-   ##
=======================================
  Coverage   27.74%   27.74%           
=======================================
  Files         192      192           
  Lines       39502    39502           
  Branches    14382    14382           
=======================================
  Hits        10959    10959           
  Misses      28151    28151           
  Partials      392      392           

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[elliottslaughter]

@rbberger Do you mind signing the commits?

[rbberger]

@elliottslaughter done