Mamba

Mamba replaces attention with a selective state space model that processes sequences in linear time, a design by Albert Gu and Tri Dao that reached transformer-level quality on language modeling where earlier subquadratic architectures fell short. The official repository provides the hardware-aware CUDA implementation, written in the spirit of FlashAttention, together with Mamba-2, which builds on the structured state space duality framework for better efficiency. Pretrained checkpoints trained on 300B tokens of the Pile are published on Hugging Face in sizes from 130M to 2.8B parameters. Installation targets Linux with Python 3.10 or newer, PyTorch, and NVIDIA GPUs on CUDA 11.6 or later, with optional precompiled kernels. Released under Apache 2.0, the code is a common starting point for researchers exploring attention-free architectures and for hybrid models that mix SSM and transformer layers, an approach several production model families have since adopted.