Don't run multi GPU test if enough GPU's aren't present #209
Conversation
|
Do you mean the ringG algorithm would fail if number of GPU < 3? One may think the ringG algorithm with non-blocking MPI isend/irecv would work even if there is only 1 MPI rank. I think it is also possible for multiple MPI ranks to share the same GPU. |
| if (DCA_HAVE_CUDA) | ||
| EXECUTE_PROCESS(COMMAND bash -c "nvidia-smi -L | awk 'BEGIN { num_gpu=0;} /GPU/ { num_gpu++;} END { printf(\"%d\", num_gpu) }'" |
There was a problem hiding this comment.
The ring algorithm should be independendent from the numbers of GPUs on the node. Rather the technical issue is that we rely on cuda aware MPI. Can we have a check similar to hostname==summit for the moment? Assuming automatically detecting if MPI is cuda aware is more complicated
There was a problem hiding this comment.
The cvd script will definitely have undefined behavior if there is not 1 GPU per rank.
There was a problem hiding this comment.
In FAQ for Open-MPI, there is mention of "Can I tell at compile time or runtime whether I have CUDA-aware support?" . It seems to test for MPIX_CUDA_AWARE_SUPPORT
https://www.open-mpi.org/faq/?category=runcuda#mpi-cuda-dev
Just FYI
PDoakORNL
dismissed
gbalduzz’s stale review
Issue added to address mpi with gpu aware dependency
this will prevent multi gpu tests from running if multiple GPU's are not present