Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Integrate the Cabana load balancer #29

Draft
wants to merge 68 commits into
base: master
Choose a base branch
from
Draft

Conversation

aetx
Copy link
Collaborator

@aetx aetx commented Aug 19, 2021

Integrating the new Cabana load balancer instead of using it directly.

aetx and others added 30 commits July 20, 2021 19:58
.github/workflows/CI.yml Outdated Show resolved Hide resolved
.github/workflows/CI.yml Outdated Show resolved Hide resolved
@aetx
Copy link
Collaborator Author

aetx commented Nov 19, 2021

On summit the loadbalanced DamBreak currently segfaults after a few iterations. This does not happen on juwels (V100), juwels-booster (A100) or my local machine.

On summit this is caused by running DamBreak using

bsub -P CSC304 -W 30 -nnodes 1 -q debug -Is /bin/bash
jsrun -n4 -c1 -g1 --smpiargs="-gpu" ./DamBreak 0.01 2 0 0.0001 1 10000 cuda

It throws the error:

terminate called after throwing an instance of 'std::runtime_error'
  what():  cudaStreamSynchronize(m_stream) error( cudaErrorIllegalAddress): an illegal memory access was encountered /ccs/home/sschulz/repos/kokkos/core/src/Cuda/Kokkos_Cuda_Instance.cpp:314
Traceback functionality not available

The error disappears for a debug build of kokkos or if the array access is checked manually in Cabana.

ExaMPM is compiled using ExaMPMCompile.txt (here as .txt since GitHub does not recognize .sh files...) The script assumes ~/repos as the current working directory and assumes that the modules cuda cmake are loaded.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants