Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix GPU clustering #205

Merged
merged 4 commits into from
Sep 14, 2023
Merged

Fix GPU clustering #205

merged 4 commits into from
Sep 14, 2023

Conversation

jakobnissen
Copy link
Member

Commit #6554009 messed up clustering on the GPU, which requires that for all array ops, both arrays are on the same device. Fix this.

NOTE: Marking as draft because currently, the .cpu() calls before torch.histogram destroys performance, so this will have to be fixed.

Commit #6554009 messed up clustering on the GPU, which requires that for all
array ops, both arrays are on the same device. Fix this.
@jakobnissen
Copy link
Member Author

Okay, this now runs as expected.

pytorch/pytorch#69519 still necessitates that we copy lots of data to and from the GPU which destroys most of the performance from using GPU. I'm quite annoyed by this but I don't see how to get around it. In older version of Vamb, we used the torch.histc function, which does run on GPU, but this can't use weights.

GPU kernels for histogram are available in https://github.com/francois-rozet/torchist, but I'm not sure we should add this dependency.

When running on a dataset with ~120k contigs, the timings are ~660s for CPU and ~812s for GPU, so here GPU is quite a bit slower.

I'm going to merge this now. I'm interested in how performance is for very large datasets (millions of contigs)

@jakobnissen jakobnissen marked this pull request as ready for review September 14, 2023 10:33
@jakobnissen jakobnissen merged commit 72f388d into master Sep 14, 2023
4 checks passed
@jakobnissen jakobnissen deleted the gpucluster branch September 14, 2023 10:33
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant