bitsandbytes

QLoRA fine-tuning on consumer GPUs exists largely because of bitsandbytes, the library that supplies 8-bit optimizers and low-bit quantization to the PyTorch ecosystem. Its block-wise quantized 8-bit optimizers match 32-bit optimizer performance while sharply cutting optimizer state memory, LLM.int8() halves the memory needed to load a model for inference, and 4-bit quantization underpins QLoRA, where low-rank adapter weights train on top of a frozen quantized base model. Hugging Face Transformers integrates it directly, which is why nearly every consumer-GPU fine-tuning workflow lists it as a dependency. Originally CUDA-only (SM60 and newer), the multi-backend effort now covers AMD ROCm on CDNA and RDNA architectures, Intel Arc and Data Center GPU Max, Intel Gaudi 2 and 3, Apple silicon on macOS 14 and later, and AVX2-capable x86 CPUs. Requirements are Python 3.10+ and PyTorch 2.4+, installed with pip install bitsandbytes. Maintained by the bitsandbytes foundation with Hugging Face support, the project is MIT licensed with about 8.4k GitHub stars.