# flash-attention pre-build wheels ![GitHub Downloads (all assets, all releases)](https://img.shields.io/github/downloads/mjun0812/flash-attention-prebuild-wheels/total?style=for-the-badge) [![Search Page](https://img.shields.io/badge/Search_Page-Find_Your_Wheel-blue?style=for-the-badge&logo=googlelens)](https://mjunya.com/flash-attention-prebuild-wheels/) ![Linux x86_64](https://img.shields.io/badge/Linux_x86_64-100.0%25-green?style=for-the-badge) ![Linux ARM64](https://img.shields.io/badge/Linux_ARM64-100.0%25-green?style=for-the-badge) ![Windows](https://img.shields.io/badge/Windows-100.0%25-green?style=for-the-badge) This repository provides wheels for the pre-built [flash-attention](https://github.com/Dao-AILab/flash-attention). Since building flash-attention takes a **very long time** and is resource-intensive, I also build and provide combinations of CUDA and PyTorch that are not officially distributed. **This repository uses a self-hosted runner and AWS CodeBuild for building the wheels. If you find this project helpful, please consider sponsoring to help maintain the infrastructure!** [![github-sponsor](https://img.shields.io/badge/sponsor-30363D?style=for-the-badge&logo=GitHub-Sponsors&logoColor=#white)](https://github.com/sponsors/mjun0812) [![buy-me-a-coffee](https://img.shields.io/badge/Buy_Me_A_Coffee-FFDD00?style=for-the-badge&logo=buy-me-a-coffee&logoColor=black)](https://buymeacoffee.com/mjun0812) **Special thanks to [@KiralyCraft](https://github.com/KiralyCraft) for providing the computing resources used to build wheels. Thank you!!** ## Install 1. Select the versions for Python, CUDA, PyTorch, and flash_attn. ```bash flash_attn-[flash_attn Version]+cu[CUDA Version]torch[PyTorch Version]-cp[Python Version]-cp[Python Version]-linux_x86_64.whl # Example: Python 3.11, CUDA 12.4, PyTorch 2.5, and flash_attn 2.6.3 flash_attn-2.6.3+cu124torch2.5-cp312-cp312-linux_x86_64.whl ``` 2. Find the corresponding version of a wheel from the **[Useful Search Page](https://mjunya.com/flash-attention-prebuild-wheels/)**, [Packages](./doc/packages.md) page, or [releases](https://github.com/mjun0812/flash-attention-prebuild-wheels/releases) page. 3. Direct Install or Download and Local Install ```bash # Direct Install pip install https://github.com/mjun0812/flash-attention-prebuild-wheels/releases/download/v0.0.0/flash_attn-2.6.3+cu124torch2.5-cp312-cp312-linux_x86_64.whl # Download and Local Install wget https://github.com/mjun0812/flash-attention-prebuild-wheels/releases/download/v0.0.0/flash_attn-2.6.3+cu124torch2.5-cp312-cp312-linux_x86_64.whl pip install ./flash_attn-2.6.3+cu124torch2.5-cp312-cp312-linux_x86_64.whl ``` ## Packages ### Coverage | Platform | Existing | Missing | Coverage | |----------|----------|---------|----------| | Linux x86_64 | 521 | 0 | 100.0% | | Linux ARM64 | 174 | 0 | 100.0% | | Windows | 159 | 0 | 100.0% | | **Total** | **854** | **0** | **100.0%** | > [!NOTE] > Since v0.8.0, Flash Attention 3 (`flash_attn_3`) wheels are also available. > Flash Attention 3 requires Hopper (SM90) or newer GPUs and CUDA 12.3+. > [!NOTE] > Since v0.7.0, wheels are built with manylinux2_28 platform. > These wheels for Linux x86_64 and ManyLinux are compatible with old glibc versions (<=2.17). > [!NOTE] > Since v0.5.0, wheels are built with a local version label indicating the CUDA and PyTorch versions. > Example: `pip list` -> `flash_attn==2.8.3 (old) -> flash_attn==2.8.3+cu130torch2.9 (>= built since v0.5.0)` See [./doc/packages.md](./doc/packages.md) for the full list of available packages. ## History The history of this repository is available [here](./doc/release_history.md). ## Citation If you use this repository in your research and find it helpful, please cite this repository! ```bibtex @misc{flash-attention-prebuild-wheels, author = {Morioka, Junya}, year = {2025}, title = {mjun0812/flash-attention-prebuild-wheels}, url = {https://github.com/mjun0812/flash-attention-prebuild-wheels}, howpublished = {https://github.com/mjun0812/flash-attention-prebuild-wheels}, } ``` ## Acknowledgments - [@okaris](https://github.com/okaris) : Sponsored me! - [@xhiroga](https://github.com/xhiroga) : Sponsored me! - cjustus613 : Buy me a coffee! - [@KiralyCraft](https://github.com/KiralyCraft) : Provided with computing resource! - [@kun432](https://github.com/kun432) : Buy me a coffee 3 times! - [@wodeyuzhou](https://github.com/wodeyuzhou) : Sponsored me! - Gabr1e1 : Buy me a coffee! - wp : Buy me a coffee! - wangxiyu191: Buy me a coffee! - [@sr99622](https://github.com/sr99622) : Sponsored me! - Zack: Buy me a coffee! ## Star History and Download Statistics Star History Chart ## Original Repository [repo](https://github.com/Dao-AILab/flash-attention) ```bibtex @inproceedings{dao2022flashattention, title={Flash{A}ttention: Fast and Memory-Efficient Exact Attention with {IO}-Awareness}, author={Dao, Tri and Fu, Daniel Y. and Ermon, Stefano and Rudra, Atri and R{\'e}, Christopher}, booktitle={Advances in Neural Information Processing Systems (NeurIPS)}, year={2022} } @inproceedings{dao2023flashattention2, title={Flash{A}ttention-2: Faster Attention with Better Parallelism and Work Partitioning}, author={Dao, Tri}, booktitle={International Conference on Learning Representations (ICLR)}, year={2024} } ``` ## Self build If you cannot find the version you are looking for, you can fork this repository and create a wheel on GitHub Actions. 1. Fork this repository 2. (Optional) Set up your self-hosted runner. 3. Edit Python script [`create_matrix.py`](https://github.com/mjun0812/flash-attention-prebuild-wheels/blob/main/create_matrix.py) to set the version you want to build. You can use GitHub hosted runners or self-hosted runners. 4. Add tag `v*.*.*` to trigger the build workflow. `git tag v*.*.* && git push --tags` Please note that depending on the combination of versions, it may not be possible to build. ### Self-Hosted Runner Build In some version combinations, you cannot build wheels on GitHub-hosted runners due to job time limitations. To build the wheels for these versions, you can use self-hosted runners. See [self-hosted-runner/README.md](./self-hosted-runner/README.md) for detailed setup instructions. ## Build Environments This repository builds wheels across multiple platforms and environments: | Platform | Runner Type | Container Image | | ------------------ | ---------------------------------- | ----------------------------------------- | | **Linux x86_64** | GitHub-hosted (`ubuntu-22.04`) | - | | **Linux x86_64** | Self-hosted | `ubuntu:24.04` or `manylinux_2_28_x86_64` | | **Linux ARM64** | GitHub-hosted (`ubuntu-22.04-arm`) | - | | **Windows x86_64** | GitHub-hosted (`windows-2022`) | - | | **Windows x86_64** | Self-hosted (`windows11`) | - | | **Windows x86_64** | AWS CodeBuild | - |