flash_attn_interface (FA3) missing from runpod/parameter-golf:latest image

[ibarrajo]

Issue

The official RunPod template (y5cejece4j, image runpod/parameter-golf:latest) does not include flash_attn_interface (Flash Attention 3). Attempting to import it fails:

>>> from flash_attn_interface import flash_attn_func
ModuleNotFoundError: No module named 'flash_attn_interface'

The standard flash_attn package (FA2) is also missing.

Impact

Multiple top submissions (#198, #254, #265) use FA3 with a bare import — no try/except fallback:

from flash_attn_interface import flash_attn_func as flash_attn_3_func

These scripts would crash on the official RunPod template. This creates confusion about the evaluation environment:

• The README says "All Python dependencies are already pre-installed in the image" and "Evaluation will be in the RunPod environment with all packages installed"
• But FA3 is not installed, and it's the attention backend used by every top submission

Environment

• Image: runpod/parameter-golf:latest
• Template: y5cejece4j
• PyTorch: 2.9.1+cu128
• CUDA: 12.8

Verified by inspecting Docker Hub layer history — no flash-attn install step exists.

Questions

1. Does the official evaluation environment differ from the RunPod template?
2. Should flash_attn_interface be added to the image?
3. Should submissions include a fallback to F.scaled_dot_product_attention for compatibility?

Workaround

For participants: use a try/except fallback to PyTorch SDPA:

try:
    from flash_attn_interface import flash_attn_func as flash_attn_3_func
except ImportError:
    def flash_attn_3_func(q, k, v, causal=True):
        q, k, v = (x.transpose(1, 2) for x in (q, k, v))
        out = F.scaled_dot_product_attention(q, k, v, is_causal=causal, enable_gqa=True)
        return out.transpose(1, 2)

This works on PyTorch 2.6+ but is ~10% slower than FA3 on H100.

[cpauldev]

I hit this too on the official RunPod template.

On a fresh pod, flash_attn_interface wasn’t installed, and flash_attn wasn’t there either.

I tried working around it by cloning Dao-AILab/flash-attention and building hopper on the pod. It did get into the real compile step, but it takes a while, which makes it costly to do on an 8xH100 pod. In my case, the connection to the host dropped before the build finished, so I wasn’t able to complete the install.

So right now the public template doesn’t seem sufficient on its own for reproducing the newer FA3-based submissions.

[pragnyanramtha]

@cpauldev , this might help you out : https://windreamer.github.io/flash-attention3-wheels/

also try downloading the data on a cheap instance to save the data on a network volume , then start your h100 instance.

[PersonalRec]

@pragnyanramtha I guess we should be trying to find the best model training params, not fix setup issues. So does that mean all those top scores will be disqualified since their scripts can't be reproduced on the RunPod setup?

[anthony-maio]

@cpauldev , this might help you out : https://windreamer.github.io/flash-attention3-wheels/

also try downloading the data on a cheap instance to save the data on a network volume , then start your h100 instance.

Hey so that wheel is built for Torch 2.8 and the pod template has 2.9.1.. I don't believe they are compatible.

[Aristotelis1]

@cpauldev , this might help you out : https://windreamer.github.io/flash-attention3-wheels/
also try downloading the data on a cheap instance to save the data on a network volume , then start your h100 instance.

Hey so that wheel is built for Torch 2.8 and the pod template has 2.9.1.. I don't believe they are compatible.

you can install the correct wheel like this:

python3 -m pip install flash_attn_3 --find-links https://windreamer.github.io/flash-attention3-wheels/cu128_torch291

[anthony-maio]

pip install flash_attn_3 --find-links https://windreamer.github.io/flash-attention3-wheels/cu128_torch291
Installs in 30 seconds. We spent ~$100 and 10+ hours building from source before discovering this. We've published the from-source build as a GitHub release for the community: https://github.com/anthony-maio/openai-parameter-golf-fa3-wheel/releases/tag/v1.0 as a backup in the name of open source competition. Thank you for your help

[kikim00]

+1, I noticed training is significantly slower(~16%) compared to FA3 and it's taking so much time to install FA3 on these expensive pods.