Skip to content

Commit

Permalink
intermediate_source/scaled_dot_product_attention_tutorial.py λ²ˆμ—­ (#773)
Browse files Browse the repository at this point in the history 
intermediate_source/scaled_dot_product_attention_tutorial.py λ²ˆμ—­
  • Loading branch information
@ganghe74
ganghe74 authored Nov 26, 2023
1 parent d1bcce9 commit c5b2847
Showing 1 changed file with 82 additions and 91 deletions.
173 changes: 82 additions & 91 deletions intermediate_source/scaled_dot_product_attention_tutorial.py
View file
Original file line number Diff line number Diff line change
@@ -1,77 +1,74 @@
"""
(Beta) Implementing High-Performance Transformers with Scaled Dot Product Attention (SDPA)
==========================================================================================
(Beta) Scaled Dot Product Attention (SDPA)둜 κ³ μ„±λŠ₯ 트랜슀포머(Transformers) κ΅¬ν˜„ν•˜κΈ°
=================================================================================
**Author:** `Driss Guessous <https://github.com/drisspg>`_
**μ €μž:** `Driss Guessous <https://github.com/drisspg>`_
**λ²ˆμ—­:** `이강희 <https://github.com/khleexv>`_
"""

######################################################################
# Summary
# ~~~~~~~~
# μš”μ•½
# ~~~~
#
# In this tutorial, we want to highlight a new ``torch.nn.functional`` function
# that can be helpful for implementing transformer architectures. The
# function is named ``torch.nn.functional.scaled_dot_product_attention``.
# For detailed description of the function, see the `PyTorch documentation <https://pytorch.org/docs/master/generated/torch.nn.functional.scaled_dot_product_attention.html#torch.nn.functional.scaled_dot_product_attention>`__.
# This function has already been incorporated into ``torch.nn.MultiheadAttention`` and ``torch.nn.TransformerEncoderLayer``.
# 이 νŠœν† λ¦¬μ–Όμ—μ„œ, 트랜슀포머(Transformer) μ•„ν‚€ν…μ²˜ κ΅¬ν˜„μ— 도움이 λ˜λŠ” μƒˆλ‘œμš΄
# ``torch.nn.functional`` λͺ¨λ“ˆμ˜ ν•¨μˆ˜λ₯Ό μ†Œκ°œν•©λ‹ˆλ‹€. 이 ν•¨μˆ˜μ˜ 이름은 ``torch.nn.functional.scaled_dot_product_attention``
# μž…λ‹ˆλ‹€. ν•¨μˆ˜μ— λŒ€ν•œ μžμ„Έν•œ μ„€λͺ…은 `PyTorch λ¬Έμ„œ <https://pytorch.org/docs/master/generated/torch.nn.functional.scaled_dot_product_attention.html#torch.nn.functional.scaled_dot_product_attention>`__
# λ₯Ό μ°Έκ³ ν•˜μ„Έμš”. 이 ν•¨μˆ˜λŠ” 이미 ``torch.nn.MultiheadAttention`` κ³Ό ``torch.nn.TransformerEncoderLayer``
# μ—μ„œ μ‚¬μš©λ˜κ³  μžˆμŠ΅λ‹ˆλ‹€.
#
# Overview
# ~~~~~~~~~
# At a high level, this PyTorch function calculates the
# scaled dot product attention (SDPA) between query, key, and value according to
# the definition found in the paper `Attention is all you
# need <https://arxiv.org/abs/1706.03762>`__. While this function can
# be written in PyTorch using existing functions, a fused implementation can provide
# large performance benefits over a naive implementation.
# κ°œμš”
# ~~~~
# κ³ μˆ˜μ€€μ—μ„œ, 이 PyTorch ν•¨μˆ˜λŠ” 쿼리(query), ν‚€(key), κ°’(value) μ‚¬μ΄μ˜
# scaled dot product attention (SDPA)을 κ³„μ‚°ν•©λ‹ˆλ‹€.
# 이 ν•¨μˆ˜μ˜ μ •μ˜λŠ” `Attention is all you need <https://arxiv.org/abs/1706.03762>`__
# λ…Όλ¬Έμ—μ„œ 찾을 수 μžˆμŠ΅λ‹ˆλ‹€. 이 ν•¨μˆ˜λŠ” κΈ°μ‘΄ ν•¨μˆ˜λ₯Ό μ‚¬μš©ν•˜μ—¬ PyTorch둜 μž‘μ„±ν•  수 μžˆμ§€λ§Œ,
# ν“¨μ¦ˆλ“œ(fused) κ΅¬ν˜„μ€ λ‹¨μˆœν•œ κ΅¬ν˜„λ³΄λ‹€ 큰 μ„±λŠ₯ 이점을 μ œκ³΅ν•  수 μžˆμŠ΅λ‹ˆλ‹€.
#
# Fused implementations
# ν“¨μ¦ˆλ“œ κ΅¬ν˜„
# ~~~~~~~~~~~~~~~~~~~~~~
#
# For CUDA tensor inputs, the function will dispatch into one of the following
# implementations:
# 이 ν•¨μˆ˜λŠ” CUDA tensor μž…λ ₯을 λ‹€μŒ 쀑 ν•˜λ‚˜μ˜ κ΅¬ν˜„μ„ μ‚¬μš©ν•©λ‹ˆλ‹€.
#
# κ΅¬ν˜„:
#
# * `FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness <https://arxiv.org/abs/2205.14135>`__
# * `Memory-Efficient Attention <https://github.com/facebookresearch/xformers>`__
# * A PyTorch implementation defined in C++
#
# .. note::
#
# This tutorial requires PyTorch 2.0.0 or later.
# 이 νŠœν† λ¦¬μ–Όμ€ PyTorch 버전 2.0.0 이상이 ν•„μš”ν•©λ‹ˆλ‹€.
#

import torch
import torch.nn as nn
import torch.nn.functional as F
device = "cuda" if torch.cuda.is_available() else "cpu"

# Example Usage:
# μ‚¬μš© μ˜ˆμ‹œ:
query, key, value = torch.randn(2, 3, 8, device=device), torch.randn(2, 3, 8, device=device), torch.randn(2, 3, 8, device=device)
F.scaled_dot_product_attention(query, key, value)


######################################################################
# Explicit Dispatcher Control
# ~~~~~~~~~~~~~~~~~~~~~~~~~~~
#
# While the function will implicitly dispatch to one of the three
# implementations, the user can also explicitly control the dispatch via
# the use of a context manager. This context manager allows users to
# explicitly disable certain implementations. If a user wants to ensure
# the function is indeed using the fastest implementation for their
# specific inputs, the context manager can be used to sweep through
# measuring performance.
# λͺ…μ‹œμ  Dispatcher μ œμ–΄
# ~~~~~~~~~~~~~~~~~~~~
#
# 이 ν•¨μˆ˜λŠ” μ•”μ‹œμ μœΌλ‘œ μ„Έ 가지 κ΅¬ν˜„ 쀑 ν•˜λ‚˜λ₯Ό μ‚¬μš©ν•©λ‹ˆλ‹€. ν•˜μ§€λ§Œ μ»¨ν…μŠ€νŠΈ λ§€λ‹ˆμ €λ₯Ό
# μ‚¬μš©ν•˜λ©΄ λͺ…μ‹œμ μœΌλ‘œ μ–΄λ–€ κ΅¬ν˜„μ„ μ‚¬μš©ν•  지 μ œμ–΄ν•  수 μžˆμŠ΅λ‹ˆλ‹€. μ»¨ν…μŠ€νŠΈ λ§€λ‹ˆμ €λ₯Ό 톡해
# νŠΉμ • κ΅¬ν˜„μ„ λͺ…μ‹œμ μœΌλ‘œ λΉ„ν™œμ„±ν™” ν•  수 μžˆμŠ΅λ‹ˆλ‹€. νŠΉμ • μž…λ ₯에 λŒ€ν•œ κ°€μž₯ λΉ λ₯Έ κ΅¬ν˜„μ„ 찾고자
# ν•œλ‹€λ©΄, μ»¨ν…μŠ€νŠΈ λ§€λ‹ˆμ €λ‘œ λͺ¨λ“  κ΅¬ν˜„μ˜ μ„±λŠ₯을 μΈ‘μ •ν•΄λ³Ό 수 μžˆμŠ΅λ‹ˆλ‹€.

# Lets define a helpful benchmarking function:
# 벀치마크 ν•¨μˆ˜λ₯Ό μ •μ˜ν•©λ‹ˆλ‹€
import torch.utils.benchmark as benchmark
def benchmark_torch_function_in_microseconds(f, *args, **kwargs):
t0 = benchmark.Timer(
stmt="f(*args, **kwargs)", globals={"args": args, "kwargs": kwargs, "f": f}
)
return t0.blocked_autorange().mean * 1e6

# Lets define the hyper-parameters of our input
# μž…λ ₯의 ν•˜μ΄νΌνŒŒλΌλ―Έν„°λ₯Ό μ •μ˜ν•©λ‹ˆλ‹€
batch_size = 32
max_sequence_len = 1024
num_heads = 32
Expand All @@ -85,7 +82,7 @@ def benchmark_torch_function_in_microseconds(f, *args, **kwargs):

print(f"The default implementation runs in {benchmark_torch_function_in_microseconds(F.scaled_dot_product_attention, query, key, value):.3f} microseconds")

# Lets explore the speed of each of the 3 implementations
# μ„Έ 가지 κ΅¬ν˜„μ˜ 속도λ₯Ό μΈ‘μ •ν•©λ‹ˆλ‹€
from torch.backends.cuda import sdp_kernel, SDPBackend

# Helpful arguments mapper
Expand Down Expand Up @@ -114,24 +111,22 @@ def benchmark_torch_function_in_microseconds(f, *args, **kwargs):


######################################################################
# Hardware dependence
# ~~~~~~~~~~~~~~~~~~~
# ν•˜λ“œμ›¨μ–΄ μ˜μ‘΄μ„±
# ~~~~~~~~~~~~~
#
# Depending on what machine you ran the above cell on and what hardware is
# available, your results might be different.
# - If you don’t have a GPU and are running on CPU then the context manager
# will have no effect and all three runs should return similar timings.
# - Depending on what compute capability your graphics card supports
# flash attention or memory efficient might have failed.
# μœ„ 셀을 μ–΄λ–€ λ¨Έμ‹ μ—μ„œ μ‹€ν–‰ν–ˆλŠ”μ§€μ™€ μ‚¬μš© κ°€λŠ₯ν•œ ν•˜λ“œμ›¨μ–΄μ— 따라 κ²°κ³Όκ°€ λ‹€λ₯Ό 수 μžˆμŠ΅λ‹ˆλ‹€.
# - GPUκ°€ μ—†κ³  CPUμ—μ„œ μ‹€ν–‰ 쀑이라면 μ»¨ν…μŠ€νŠΈ λ§€λ‹ˆμ €λŠ” νš¨κ³Όκ°€ μ—†κ³  μ„Έ 가지 μ‹€ν–‰ λͺ¨λ‘
# μœ μ‚¬ν•œ μ‹œκ°„μ„ λ°˜ν™˜ν•  κ²ƒμž…λ‹ˆλ‹€.
# - κ·Έλž˜ν”½ μΉ΄λ“œκ°€ μ§€μ›ν•˜λŠ” μ»΄ν“¨νŒ… λŠ₯λ ₯에 따라 flash attention λ˜λŠ”
# memory efficient κ΅¬ν˜„μ΄ λ™μž‘ν•˜μ§€ μ•Šμ„ 수 μžˆμŠ΅λ‹ˆλ‹€.


######################################################################
# Causal Self Attention
# ~~~~~~~~~~~~~~~~~~~~~
#
# Below is an example implementation of a multi-headed causal self
# attention block inspired by
# `Andrej Karpathy NanoGPT <https://github.com/karpathy/nanoGPT>`__ repository.
# μ•„λž˜λŠ” multi-head causal self attention λΈ”λ‘μ˜ κ΅¬ν˜„ μ˜ˆμ‹œμž…λ‹ˆλ‹€.
# `Andrej Karpathy NanoGPT <https://github.com/karpathy/nanoGPT>`__ μ €μž₯μ†Œλ₯Ό μ°Έκ³ ν–ˆμŠ΅λ‹ˆλ‹€.
#

class CausalSelfAttention(nn.Module):
Expand Down Expand Up @@ -187,12 +182,13 @@ def forward(self, x):


#####################################################################
# ``NestedTensor`` and Dense tensor support
# -----------------------------------------
# ``NestedTensor`` 및 Dense tensor 지원
# ------------------------------------
#
# SDPA supports both ``NestedTensor`` and Dense tensor inputs. ``NestedTensors`` handle the case where the input is a batch of variable length sequences
# without needing to pad each sequence to the maximum length in the batch. For more information about ``NestedTensors`` see
# `torch.nested <https://pytorch.org/docs/stable/nested.html>`__ and `NestedTensors Tutorial <https://tutorials.pytorch.kr/prototype/nestedtensor.html>`__.
# SDPAλŠ” ``NestedTensor`` 와 Dense tensor μž…λ ₯을 λͺ¨λ‘ μ§€μ›ν•©λ‹ˆλ‹€.
# ``NestedTensors`` λŠ” μž…λ ₯이 κ°€λ³€ 길이 μ‹œν€€μŠ€λ‘œ κ΅¬μ„±λœ 배치인 κ²½μš°μ—
# 배치 λ‚΄ μ‹œν€€μŠ€μ˜ μ΅œλŒ€ 길이에 맞좰 각 μ‹œν€€μŠ€λ₯Ό νŒ¨λ”©ν•  ν•„μš”κ°€ μ—†μŠ΅λ‹ˆλ‹€. ``NestedTensors`` 에 λŒ€ν•œ μžμ„Έν•œ λ‚΄μš©μ€
# `torch.nested <https://pytorch.org/docs/stable/nested.html>`__ 와 `NestedTensors νŠœν† λ¦¬μ–Ό <https://tutorials.pytorch.kr/prototype/nestedtensor.html>`__ 을 μ°Έκ³ ν•˜μ„Έμš”.
#

import random
Expand Down Expand Up @@ -236,7 +232,7 @@ def generate_rand_batch(
random_nt, _ = generate_rand_batch(32, 512, embed_dimension, pad_percentage=0.5, dtype=dtype, device=device)
random_dense, _ = generate_rand_batch(32, 512, embed_dimension, pad_percentage=None, dtype=dtype, device=device)

# Currently the fused implementations don't support ``NestedTensor`` for training
# ν˜„μž¬ ν“¨μ¦ˆλ“œ κ΅¬ν˜„μ€ ``NestedTensor`` 둜 ν•™μŠ΅ν•˜λŠ” 것을 μ§€μ›ν•˜μ§€ μ•ŠμŠ΅λ‹ˆλ‹€.
model.eval()

with sdp_kernel(**backend_map[SDPBackend.FLASH_ATTENTION]):
Expand All @@ -248,15 +244,14 @@ def generate_rand_batch(


######################################################################
# Using SDPA with ``torch.compile``
# =================================
# ``torch.compile`` κ³Ό ν•¨κ»˜ SDPA μ‚¬μš©ν•˜κΈ°
# =====================================
#
# With the release of PyTorch 2.0, a new feature called
# ``torch.compile()`` has been introduced, which can provide
# significant performance improvements over eager mode.
# Scaled dot product attention is fully composable with ``torch.compile()``.
# To demonstrate this, let's compile the ``CausalSelfAttention`` module using
# ``torch.compile()`` and observe the resulting performance improvements.
# PyTorch 2.0 λ¦΄λ¦¬μ¦ˆμ™€ ν•¨κ»˜ ``torch.compile()`` λΌλŠ” μƒˆλ‘œμš΄ κΈ°λŠ₯이 μΆ”κ°€λ˜μ—ˆλŠ”λ°,
# μ΄λŠ” eager mode보닀 μƒλ‹Ήν•œ μ„±λŠ₯ ν–₯상을 μ œκ³΅ν•  수 μžˆμŠ΅λ‹ˆλ‹€.
# Scaled dot product attention은 ``torch.compile()`` 둜 μ™„μ „νžˆ ꡬ성할 수 μžˆμŠ΅λ‹ˆλ‹€.
# 이λ₯Ό ν™•μΈν•˜κΈ° μœ„ν•΄ ``torch.compile()`` 을 톡해 ``CausalSelfAttention`` λͺ¨λ“ˆμ„ μ»΄νŒŒμΌν•˜κ³ 
# 결과적으둜 μ–»μ–΄μ§€λŠ” μ„±λŠ₯ ν–₯상을 μ•Œμ•„λ΄…μ‹œλ‹€.
#

batch_size = 32
Expand All @@ -276,12 +271,11 @@ def generate_rand_batch(

######################################################################
#
# The exact execution time is dependent on machine, however the results for mine:
# The non compiled module runs in 166.616 microseconds
# The compiled module runs in 166.726 microseconds
# That is not what we were expecting. Let's dig a little deeper.
# PyTorch comes with an amazing built-in profiler that you can use to
# inspect the performance characteristics of your code.
# μ •ν™•ν•œ μ‹€ν–‰ μ‹œκ°„μ€ ν™˜κ²½μ— 따라 λ‹€λ₯΄μ§€λ§Œ, λ‹€μŒμ€ μ €μžμ˜ κ²°κ³Όμž…λ‹ˆλ‹€.
# 컴파일 λ˜μ§€ μ•Šμ€ λͺ¨λ“ˆμ€ 싀행에 166.616ms κ°€ μ†Œμš”λ˜μ—ˆμŠ΅λ‹ˆλ‹€.
# 컴파일 된 λͺ¨λ“ˆμ€ 싀행에 166.726ms κ°€ μ†Œμš”λ˜μ—ˆμŠ΅λ‹ˆλ‹€.
# μ΄λŠ” 우리의 μ˜ˆμƒκ³ΌλŠ” λ‹€λ¦…λ‹ˆλ‹€. μ’€ 더 μžμ„Ένžˆ μ•Œμ•„λ΄…μ‹œλ‹€.
# PyTorchλŠ” μ½”λ“œμ˜ μ„±λŠ₯ νŠΉμ„±μ„ 점검할 수 μžˆλŠ” λ†€λΌμš΄ λ‚΄μž₯(built-in) ν”„λ‘œνŒŒμΌλŸ¬λ₯Ό μ œκ³΅ν•©λ‹ˆλ‹€.
#

from torch.profiler import profile, record_function, ProfilerActivity
Expand All @@ -302,7 +296,7 @@ def generate_rand_batch(
compiled_model(x)
print(prof.key_averages().table(sort_by="cuda_time_total", row_limit=10))

# For even more insights, you can export the trace and use ``chrome://tracing`` to view the results
# 더 λ§Žμ€ 정보λ₯Ό μ–»κΈ° μœ„ν•΄ 좔적(trace)λ₯Ό 내보내고 ``chrome://tracing``을 μ‚¬μš©ν•˜μ—¬ κ²°κ³Όλ₯Ό ν™•μΈν•΄λ³΄μ„Έμš”.
# ::
#
# prof.export_chrome_trace("compiled_causal_attention_trace.json").
Expand All @@ -311,33 +305,30 @@ def generate_rand_batch(


######################################################################
# The previous code snippet generates a report of the top 10 PyTorch functions
# that consumed the most GPU execution time, for both the compiled and non-compiled module.
# The analysis reveals that the majority of time spent on the GPU is concentrated
# on the same set of functions for both modules.
# The reason for this here is that ``torch.compile`` is very good at removing the
# framework overhead associated with PyTorch. If your model is launching
# large, efficient CUDA kernels, which in this case ``CausaulSelfAttention``
# is, then the overhead of PyTorch can be hidden.
# 이전 μ½”λ“œ 쑰각(snippet)은 컴파일 된 λͺ¨λ“ˆκ³Ό μ»΄νŒŒμΌλ˜μ§€ μ•Šμ€ λͺ¨λ“ˆ λͺ¨λ‘μ— λŒ€ν•΄
# κ°€μž₯ λ§Žμ€ GPU μ‹€ν–‰ μ‹œκ°„μ„ μ°¨μ§€ν•œ μƒμœ„ 10개의 PyTorch ν•¨μˆ˜μ— λŒ€ν•œ λ³΄κ³ μ„œλ₯Ό μƒμ„±ν•©λ‹ˆλ‹€.
# 뢄석 κ²°κ³Ό, 두 λͺ¨λ“ˆ λͺ¨λ‘ GPUμ—μ„œ μ†Œμš”λœ μ‹œκ°„μ˜ λŒ€λΆ€λΆ„μ΄
# λ™μΌν•œ ν•¨μˆ˜λ“€μ— μ§‘μ€‘λ˜μ–΄ μžˆμŒμ„ λ³΄μ—¬μ€λ‹ˆλ‹€.
# PyTorchκ°€ ν”„λ ˆμž„μ›Œν¬ μ˜€λ²„ν—€λ“œλ₯Ό μ œκ±°ν•˜λŠ” 데 맀우 νƒμ›”ν•œ ``torch.compile`` λ₯Ό
# μ œκ³΅ν•˜κΈ° λ•Œλ¬Έμž…λ‹ˆλ‹€. ``CausalSelfAttention`` 같은 경우처럼 크고, 효율적인 CUDA 컀널을
# μ‚¬μš©ν•˜λŠ” λͺ¨λΈμ—μ„œ PyTorch μ˜€λ²„ν—€λ“œλŠ” μž‘μ•„μ§ˆ κ²ƒμž…λ‹ˆλ‹€.
#
# In reality, your module does not normally consist of a singular
# ``CausalSelfAttention`` block. When experimenting with `Andrej Karpathy NanoGPT <https://github.com/karpathy/nanoGPT>`__ repository, compiling
# the module took the time per train step from: ``6090.49ms`` to
# ``3273.17ms``! This was done on commit: ``ae3a8d5`` of NanoGPT training on
# the Shakespeare dataset.
# 사싀, λͺ¨λ“ˆμ€ 보톡 ``CausalSelfAttention`` λΈ”λŸ­ ν•˜λ‚˜λ§ŒμœΌλ‘œ κ΅¬μ„±λ˜μ§€ μ•ŠμŠ΅λ‹ˆλ‹€.
# `Andrej Karpathy NanoGPT <https://github.com/karpathy/nanoGPT>`__ μ €μž₯μ†Œμ—μ„œ μ‹€ν—˜ν•œ 경우,
# λͺ¨λ“ˆμ„ 컴파일 ν•˜λŠ” 것은 ν•™μŠ΅μ˜ 각 단계별 μ†Œμš” μ‹œκ°„μ„ ``6090.49ms`` μ—μ„œ ``3273.17ms`` 둜
# 쀄일 수 μžˆμ—ˆμŠ΅λ‹ˆλ‹€. 이 μ‹€ν—˜μ€ NanoGPT μ €μž₯μ†Œμ˜ ``ae3a8d5`` μ»€λ°‹μ—μ„œ Shakespeare
# 데이터셋을 μ‚¬μš©ν•˜μ—¬ μ§„ν–‰λ˜μ—ˆμŠ΅λ‹ˆλ‹€.
#


######################################################################
# Conclusion
# ==========
# κ²°λ‘ 
# ====
#
# In this tutorial, we have demonstrated the basic usage of
# ``torch.nn.functional.scaled_dot_product_attention``. We have shown how
# the ``sdp_kernel`` context manager can be used to assert a certain
# implementation is used on GPU. As well, we built a simple
# ``CausalSelfAttention`` module that works with ``NestedTensor`` and is torch
# compilable. In the process we have shown how to the profiling tools can
# be used to explore the performance characteristics of a user defined
# module.
# 이 νŠœν† λ¦¬μ–Όμ—μ„œ, ``torch.nn.functional.scaled_dot_product_attention`` 의 기본적인
# μ‚¬μš©λ²•μ„ μ‚΄νŽ΄λ΄€μŠ΅λ‹ˆλ‹€. ``sdp_kernel`` μ»¨ν…μŠ€νŠΈ λ§€λ‹ˆμ €λ‘œ GPUκ°€ νŠΉμ • κ΅¬ν˜„μ„
# μ‚¬μš©ν•˜λ„λ‘ ν•  수 μžˆλ‹€λŠ” 것을 λ³΄μ•˜μŠ΅λ‹ˆλ‹€. λ˜ν•œ, κ°„λ‹¨ν•œ ``NestedTensor`` μ—μ„œ μž‘λ™ν•˜κ³ 
# 컴파일 κ°€λŠ₯ν•œ ``CausalSelfAttention`` λͺ¨λ“ˆμ„ λ§Œλ“€μ—ˆμŠ΅λ‹ˆλ‹€.
# 이 κ³Όμ •μ—μ„œ ν”„λ‘œνŒŒμΌλ§ 도ꡬλ₯Ό μ‚¬μš©ν•˜μ—¬ μœ μ €κ°€ μ •μ˜ν•œ λͺ¨λ“ˆμ˜ μ„±λŠ₯ νŠΉμ„±μ„ μ–΄λ–»κ²Œ
# 확인할 수 μžˆλŠ”μ§€λ„ μ‚΄νŽ΄λ΄€μŠ΅λ‹ˆλ‹€.
#

0 comments on commit c5b2847

Please sign in to comment.