Very long JIT times

[abagusetty]

Point of Contact

Abhi, Colleen

Contact Details

abagusetty@anl.gov

Vendor/ALCF/other tickets/IDs

GSD-12462

Reproducer Path

https://github.com/wavefunction91/ExchCXX and source/reproducers/dpcpp/very_long_jit

Status

Open

Details

https://github.com/wavefunction91/ExchCXX

Todo: get CUDA timings

Is this a priority/blocking bug?

• Priority

ETA

sycl default subgroup size 16 helps, still under investigation

[abagusetty]

Adding some details on CUDA build times for H100 (sm_90):

cmake .. -DEXCHCXX_ENABLE_CUDA=ON -DCMAKE_CUDA_ARCHITECTURES=90 -DCMAKE_C_COMPILER=`which gcc` -DCMAKE_CXX_COMPILER=`which g++`
time make -j16
...
real	8m20.209s
user	16m49.613s
sys	0m24.023s

[abagusetty]

To clarify: the above build for 8m20.209s is a AoT build for sm_90. I checked the binary with cuobjdump. It contains embedded sm_90 ELF/SASS, so it likely generated AoT.

I did also try this to enforce a PTX-only build instead:

cmake -S . -B build-jit \
  -DEXCHCXX_ENABLE_CUDA=ON \
  -DCMAKE_CUDA_ARCHITECTURES=90-virtual \
  -DCMAKE_C_COMPILER=$(which gcc) \
  -DCMAKE_CXX_COMPILER=$(which g++) \
  -DCMAKE_BUILD_TYPE=Release

time cmake --build build-jit -j16
...
real	6m32.516s
user	22m17.304s
sys	0m46.020s

The above time is PTX-only compile time. Not sure what is the equivalent of CUDA's JIT only build times. The H100 JIT cost is the first-run latency of that PTX-only executable and the ~8m20s vs ~6m32s difference reflects the added AoT cost of emitting native sm_90 code.