Skip to content

Commit

Permalink
beginner_source/ddp_series_intro.rst PyTorchKorea#891
Browse files Browse the repository at this point in the history
  • Loading branch information
hojunking committed Aug 25, 2024
1 parent 857c4b7 commit 5c07626
Showing 1 changed file with 31 additions and 46 deletions.
77 changes: 31 additions & 46 deletions beginner_source/ddp_series_intro.rst
Original file line number Diff line number Diff line change
@@ -1,56 +1,41 @@
**Introduction** \|\| `What is DDP <ddp_series_theory.html>`__ \|\|
`Single-Node Multi-GPU Training <ddp_series_multigpu.html>`__ \|\|
`Fault Tolerance <ddp_series_fault_tolerance.html>`__ \|\|
`Multi-Node training <../intermediate/ddp_series_multinode.html>`__ \|\|
`minGPT Training <../intermediate/ddp_series_minGPT.html>`__
**์†Œ๊ฐœ** \|\| `DDP๋ž€ ๋ฌด์—‡์ธ๊ฐ€ <ddp_series_theory.html>`__ \|\|
`๋‹จ์ผ ๋…ธ๋“œ ๋‹ค์ค‘-GPU ํ•™์Šต <ddp_series_multigpu.html>`__ \|\|
`์žฅ์•  ๋‚ด์„ฑ <ddp_series_fault_tolerance.html>`__ \|\|
`๋‹ค์ค‘ ๋…ธ๋“œ ํ•™์Šต <../intermediate/ddp_series_multinode.html>`__ \|\|
`minGPT ํ•™์Šต <../intermediate/ddp_series_minGPT.html>`__

Distributed Data Parallel in PyTorch - Video Tutorials
======================================================
PyTorch์˜ ๋ถ„์‚ฐ ๋ฐ์ดํ„ฐ ๋ณ‘๋ ฌ ์ฒ˜๋ฆฌ - ๋น„๋””์˜ค ํŠœํ† ๋ฆฌ์–ผ
=====================================================

Authors: `Suraj Subramanian <https://github.com/suraj813>`__
์ €์ž: `Suraj Subramanian <https://github.com/suraj813>`__

Follow along with the video below or on `youtube <https://www.youtube.com/watch/-K3bZYHYHEA>`__.
์•„๋ž˜ ๋น„๋””์˜ค๋ฅผ ๋ณด๊ฑฐ๋‚˜ `YouTube <https://www.youtube.com/watch/-K3bZYHYHEA>`__์—์„œ ํ•จ๊ป˜ ์‹œ์ฒญํ•˜์„ธ์š”.

.. raw:: html

<div style="margin-top:10px; margin-bottom:10px;">
<iframe width="560" height="315" src="https://www.youtube.com/embed/-K3bZYHYHEA" frameborder="0" allow="accelerometer; encrypted-media; gyroscope; picture-in-picture" allowfullscreen></iframe>
</div>

This series of video tutorials walks you through distributed training in
PyTorch via DDP.

The series starts with a simple non-distributed training job, and ends
with deploying a training job across several machines in a cluster.
Along the way, you will also learn about
`torchrun <https://pytorch.org/docs/stable/elastic/run.html>`__ for
fault-tolerant distributed training.

The tutorial assumes a basic familiarity with model training in PyTorch.

Running the code
----------------

You will need multiple CUDA GPUs to run the tutorial code. Typically,
this can be done on a cloud instance with multiple GPUs (the tutorials
use an Amazon EC2 P3 instance with 4 GPUs).

The tutorial code is hosted in this
`github repo <https://github.com/pytorch/examples/tree/main/distributed/ddp-tutorial-series>`__.
Clone the repository and follow along!

Tutorial sections
-----------------

0. Introduction (this page)
1. `What is DDP? <ddp_series_theory.html>`__ Gently introduces what DDP is doing
under the hood
2. `Single-Node Multi-GPU Training <ddp_series_multigpu.html>`__ Training models
using multiple GPUs on a single machine
3. `Fault-tolerant distributed training <ddp_series_fault_tolerance.html>`__
Making your distributed training job robust with torchrun
4. `Multi-Node training <../intermediate/ddp_series_multinode.html>`__ Training models using
multiple GPUs on multiple machines
5. `Training a GPT model with DDP <../intermediate/ddp_series_minGPT.html>`__ โ€œReal-worldโ€
example of training a `minGPT <https://github.com/karpathy/minGPT>`__
model with DDP
์ด ๋น„๋””์˜ค ํŠœํ† ๋ฆฌ์–ผ ์‹œ๋ฆฌ์ฆˆ๋Š” PyTorch์—์„œ DDP(Distributed Data Parallel)๋ฅผ ์‚ฌ์šฉํ•œ ๋ถ„์‚ฐ ํ•™์Šต์— ๋Œ€ํ•ด ์•ˆ๋‚ดํ•ฉ๋‹ˆ๋‹ค.

์ด ์‹œ๋ฆฌ์ฆˆ๋Š” ๋‹จ์ˆœํ•œ ๋น„๋ถ„์‚ฐ ํ•™์Šต ์ž‘์—…์—์„œ ์‹œ์ž‘ํ•˜์—ฌ, ํด๋Ÿฌ์Šคํ„ฐ ๋‚ด ์—ฌ๋Ÿฌ ๊ธฐ๊ธฐ๋“ค(multiple machines)์—์„œ ํ•™์Šต ์ž‘์—…์„ ๋ฐฐํฌํ•˜๋Š” ๊ฒƒ์œผ๋กœ ๋งˆ๋ฌด๋ฆฌ๋ฉ๋‹ˆ๋‹ค. ์ด ๊ณผ์ •์—์„œ `torchrun <https://pytorch.org/docs/stable/elastic/run.html>`__์„ ์‚ฌ์šฉํ•œ ์žฅ์•  ํ—ˆ์šฉ(fault-tolerant) ๋ถ„์‚ฐ ํ•™์Šต์— ๋Œ€ํ•ด์„œ๋„ ๋ฐฐ์šฐ๊ฒŒ ๋ฉ๋‹ˆ๋‹ค.

์ด ํŠœํ† ๋ฆฌ์–ผ์€ PyTorch์—์„œ ๋ชจ๋ธ ํ•™์Šต์— ๋Œ€ํ•œ ๊ธฐ๋ณธ์ ์ธ ์ดํ•ด๋ฅผ ์ „์ œ๋กœ ํ•ฉ๋‹ˆ๋‹ค.

์ฝ”๋“œ ์‹คํ–‰
--------

ํŠœํ† ๋ฆฌ์–ผ ์ฝ”๋“œ๋ฅผ ์‹คํ–‰ํ•˜๋ ค๋ฉด ์—ฌ๋Ÿฌ ๊ฐœ์˜ CUDA GPU๊ฐ€ ํ•„์š”ํ•ฉ๋‹ˆ๋‹ค. ์ผ๋ฐ˜์ ์œผ๋กœ ์—ฌ๋Ÿฌ GPU๊ฐ€ ์žˆ๋Š” ํด๋ผ์šฐ๋“œ ์ธ์Šคํ„ด์Šค์—์„œ ์ด๋ฅผ ์ˆ˜ํ–‰ํ•  ์ˆ˜ ์žˆ์œผ๋ฉฐ, ํŠœํ† ๋ฆฌ์–ผ์—์„œ๋Š” 4๊ฐœ์˜ GPU๊ฐ€ ํƒ‘์žฌ๋œ Amazon EC2 P3 ์ธ์Šคํ„ด์Šค๋ฅผ ์‚ฌ์šฉํ•ฉ๋‹ˆ๋‹ค.

ํŠœํ† ๋ฆฌ์–ผ ์ฝ”๋“œ๋Š” ์ด `GitHub ์ €์žฅ์†Œ <https://github.com/pytorch/examples/tree/main/distributed/ddp-tutorial-series>`__์— ํ˜ธ์ŠคํŒ…๋˜์–ด ์žˆ์Šต๋‹ˆ๋‹ค. ์ €์žฅ์†Œ๋ฅผ ๋ณต์ œํ•˜๊ณ  ํ•จ๊ป˜ ์ง„ํ–‰ํ•˜์„ธ์š”!

ํŠœํ† ๋ฆฌ์–ผ ์„น์…˜
--------------

0. ์†Œ๊ฐœ (์ด ํŽ˜์ด์ง€)
1. `DDP๋ž€ ๋ฌด์—‡์ธ๊ฐ€? <ddp_series_theory.html>`__ DDP๊ฐ€ ๋‚ด๋ถ€์ ์œผ๋กœ ์ˆ˜ํ–‰ํ•˜๋Š” ์ž‘์—…์— ๋Œ€ํ•ด ๊ฐ„๋‹จํžˆ ์†Œ๊ฐœํ•ฉ๋‹ˆ๋‹ค.
2. `์‹ฑ๊ธ€ ๋…ธ๋“œ ๋ฉ€ํ‹ฐ-GPU ํ•™์Šต <ddp_series_multigpu.html>`__ ํ•œ ๊ธฐ๊ธฐ์—์„œ ์—ฌ๋Ÿฌ GPU๋ฅผ ์‚ฌ์šฉํ•˜์—ฌ ๋ชจ๋ธ์„ ํ•™์Šตํ•˜๋Š” ๋ฐฉ๋ฒ•
3. `์žฅ์•  ๋‚ด์„ฑ ๋ถ„์‚ฐ ํ•™์Šต <ddp_series_fault_tolerance.html>`__ torchrun์„ ์‚ฌ์šฉํ•˜์—ฌ ๋ถ„์‚ฐ ํ•™์Šต ์ž‘์—…์„ ๊ฒฌ๊ณ ํ•˜๊ฒŒ ๋งŒ๋“œ๋Š” ๋ฐฉ๋ฒ•
4. `๋ฉ€ํ‹ฐ ๋…ธ๋“œ ํ•™์Šต <../intermediate/ddp_series_multinode.html>`__ ์—ฌ๋Ÿฌ ๊ธฐ๊ธฐ์—์„œ ์—ฌ๋Ÿฌ GPU๋ฅผ ์‚ฌ์šฉํ•˜์—ฌ ๋ชจ๋ธ์„ ํ•™์Šตํ•˜๋Š” ๋ฐฉ๋ฒ•
5. `DDP๋ฅผ ์‚ฌ์šฉํ•œ GPT ๋ชจ๋ธ ํ•™์Šต <../intermediate/ddp_series_minGPT.html>`__ DDP๋ฅผ ์‚ฌ์šฉํ•œ `minGPT <https://github.com/karpathy/minGPT>`__ ๋ชจ๋ธ ํ•™์Šต์˜ โ€œ์‹ค์ œ ์˜ˆ์‹œโ€

0 comments on commit 5c07626

Please sign in to comment.