fix: use num_stages=1 on ROCm to avoid Triton pipelining crash on RDNA4 (gfx12xx)#365
Open
brosequist wants to merge 1 commit into
Open
fix: use num_stages=1 on ROCm to avoid Triton pipelining crash on RDNA4 (gfx12xx)#365brosequist wants to merge 1 commit into
brosequist wants to merge 1 commit into
Conversation
…A4 (gfx12xx) Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
|
Hello @jt-zhang @whx1003 @Weiqi-Feng, I am from AMD and we verified that OP's commit fixes the issue on gfx12xx GPUs. Is there a possibility to merge this PR? Also, is there a possibility to have this included in the v1.0.6? |
3 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
On AMD RDNA4 GPUs (gfx1200 / gfx1201, e.g. RX 9060 XT / RX 9070 XT) with ROCm 7.2.2 and Triton 3.6.0, the sageattention Triton kernels fail immediately on first use with:
Full traceback snippet:
Root Cause
The Triton AMD software pipelining pass (
tritonamdgpu-pipeline) has a use-after-free bug triggered whennum_stages >= 2on gfx12xx targets. RDNA4 (gfx1200/gfx1201) is not in Triton 3.6.0's verified target list, and the pipelining pass does not handle these architectures correctly.Fix
Use
num_stages=1when running on ROCm.torch.version.hipis a non-Nonestring on ROCm builds andNoneon CUDA builds, making it a reliable runtime discriminator with no import overhead.num_stages=1disables software pipelining entirely, which avoids the bug. CUDA behavior is completely unchanged.Testing
Tested on:
With this fix, all 4 Triton kernel files compile and run correctly on RDNA4. Attention quality is unaffected;
num_stages=1only disables prefetching, not correctness.Notes
*_h96.pyvariant files (head_dim=96) contain the samenum_stages=4pattern and may need the same fix if present in other installed versions — they were absent from the upstream source but appeared in the pip-installed 1.0.6 package.num_stages=1on all ROCm targets is safe and simple.