About masked acceleration

[Harry-Miral]

Can I still use SageAttention to speed up if I manually provide att_mask (not causal nor padding mask)？Thank you.

[jt-zhang]

Hi, I think SpargeAttn may have done the thing you mentioned: https://github.com/thu-ml/SpargeAttn.

[Harry-Miral]

BUT.
After reading the code again, I only found the causal mask.

[jt-zhang]

Hi, Have you read the SpargeAttn code? Please note that it is different from SageAttention.

[Harry-Miral]

You are a genius! I love you! I confused the two things, they are so similar.

[Harry-Miral]

Also, does it support different length data within the same batch?

[jason-huang03]

@Harry-Miral Currently not. What kind of workload are you working on? If the batch size is quite small, then you can use for loop on each single batch of data.