Repository navigation
Conversation
With cuDNN frontend canonical BF16 layouts (supports_bf16_canonical_layouts), the FC1 dgrad grouped GEMM and the FC2 dGLU take the stored (experts, out, in) weight directly with b_major="n": no per-backward transpose copy for FC1 and no pointer array for FC2. Signed-off-by: Hwanseo Choi <hwanseoc@nvidia.com>
make_grouped_tensor_with_shapes builds per-tensor split metadata on every backward (about 217 us per call); the uniform-shape unquantized buffer only needs make_grouped_tensor_from_rowwise_data over one allocation. Signed-off-by: Hwanseo Choi <hwanseoc@nvidia.com>
8 of 13 tasks
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Staging PR in the fork, stacked on @tingyangk's BF16 fused grouped MLP (NVIDIA#3646).
hwanseoc/te3646-baseis that PR's commit.7f47508): when the cuDNN frontend advertisessupports_bf16_canonical_layouts(Accept canonical layouts in the BF16 grouped GEMM backends and stop reading offsets to the host NVIDIA/cudnn-frontend#1602, merged), a packed weight goes to cuDNN as the stored(experts, out, in)buffer withb_major="n". FC1 dgrad no longer transposes the whole weight every step, and FC2 dGLU no longer builds a pointer array.46f5919): the buffer is allocated directly withmake_grouped_tensor_from_rowwise_datainstead ofmake_grouped_tensor_with_shapes, which cost about 217 µs per call. This function is shared, so the MXFP8 and NVFP4 paths get it too.Measured on B200 against NVIDIA#3646 with cuDNN FE develop:
@coderabbitai ignore