xFormers

Memory-efficient attention was popularized by xFormers, Meta's toolbox of optimized, composable transformer building blocks that predates and still complements PyTorch's built-in scaled dot product attention. The library provides exact attention kernels the maintainers measure at up to 10x faster than naive implementations, block-sparse and sparse attention variants, and fused operations for softmax, linear layers, layer norm, SwiGLU, and dropout-activation chains, all exposed as independent components that slot into existing PyTorch models. It remains a default acceleration dependency across the ecosystem: Stable Diffusion tooling, LLM fine-tuning stacks, and inference servers list it for its attention speed and memory savings. Prebuilt wheels install via pip alongside recent PyTorch on Linux and Windows with CUDA 12.6 through 13.0, experimental ROCm 7.1 wheels cover AMD hardware on Linux, and building from source needs a full CUDA toolchain. The code is BSD-3-Clause licensed and holds about 10,500 GitHub stars, and research teams use its components to prototype new attention mechanisms without writing Triton or CUDA from scratch.