New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[performance doc] add tables with DDP/DP/no-DP benchmarks #13925

Open

stas00 opened this issue Oct 7, 2021 · 0 comments

Labels

WIP

Contributor

stas00 commented Oct 7, 2021 •

edited

Loading

Users ask of DDP vs. DP vs. single GPU vs w/ w/o Nvlink and this summary is insufficient
https://huggingface.co/transformers/performance.html#nvlink
luckily we have great numbers posted - so should link to those from the guide:

table 1 where DDP wins by miles over any other setup: Excessive GPU-GPU communication with GPT2 making multi-GPU training slow? #9371 (comment)
table 2 where DDP is terrible Excessive GPU-GPU communication with GPT2 making multi-GPU training slow? #9371 (comment)

stas00 self-assigned this

stas00 removed their assignment

stas00 added the WIP label

huggingface deleted a comment from github-actions bot

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment