flash-attn PyTorch Compatibility
Check which PyTorch versions work with flash-attn. PyTorch 2.4-2.5 are recommended for the best wheel availability. PyTorch 2.9 is supported in flash-attn 2.8.3+.
Quick Answer: PyTorch Compatibility
- PyTorch 2.9: Use flash-attn 2.8.3+
- PyTorch 2.5-2.8: Use flash-attn 2.6.3+ (recommended)
- PyTorch 2.4: Use flash-attn 2.5.9+
- PyTorch 2.0-2.3: Use flash-attn 2.5.9 or older
Check Your PyTorch Version
Run this command to see your PyTorch version and CUDA build:
python -c "import torch; print(f'PyTorch: {torch.__version__}, CUDA: {torch.version.cuda}')"The CUDA version in PyTorch must match the wheel you download. For example, if PyTorch reports CUDA 12.1, download a flash-attn wheel built for CUDA 12.1.
Detailed PyTorch Compatibility
| PyTorch | Status | flash-attn Versions | CUDA |
|---|---|---|---|
| 2.9 | supported | 2.8.3+ | 12.4, 12.6, 12.8 |
| 2.8 | supported | 2.8.0+ | 12.1, 12.4, 12.6 |
| 2.7 | supported | 2.7.4+ | 12.1, 12.4 |
| 2.6 | supported | 2.7.0+ | 12.1, 12.4 |
| 2.5 | recommended | 2.6.3+ | 11.8, 12.1, 12.4 |
| 2.4 | recommended | 2.5.9+ | 11.8, 12.1, 12.4 |
| 2.3 | supported | 2.4.0 - 2.6.3 | 11.8, 12.1 |
| 2.2 | legacy | 2.3.0 - 2.5.9 | 11.8, 12.1 |
| 2.1 | legacy | 2.2.0 - 2.5.9 | 11.8, 12.1 |
| 2.0 | legacy | 2.0.0 - 2.5.9 | 11.7, 11.8 |
Common PyTorch Compatibility Issues
ImportError: undefined symbol
This usually means your flash-attn wheel was built for a different PyTorch version. Check that your installed PyTorch exactly matches the wheel's expected version.
ABI mismatch
PyTorch versions can have different C++ ABIs. If you see ABI-related errors, try a wheel with a different CXX11 ABI setting, or use a different flash-attn version.
CUDA version mismatch
The CUDA version in your PyTorch build must match the flash-attn wheel. Check with torch.version.cuda.