LoRAX

Serving one fine-tune per GPU gets expensive fast, and LoRAX exists to fix that: Predibase's inference server, derived from Hugging Face's text-generation-inference, dynamically loads LoRA adapters from Hugging Face, Predibase, or local files and batches requests for thousands of different adapters on a single GPU. Heterogeneous continuous batching mixes adapters inside one batch without stalling throughput, while adapter exchange scheduling pages adapters between GPU and CPU memory asynchronously. The stack includes tensor parallelism, pre-compiled CUDA kernels with flash attention and paged attention, quantization via bitsandbytes, GPTQ, and AWQ, an OpenAI-compatible API with multi-turn chat, and production tooling: Docker images, Kubernetes Helm charts, Prometheus metrics, and OpenTelemetry tracing. Running it requires Linux, an NVIDIA GPU of Ampere generation or newer, and CUDA 11.8 compatible drivers, typically launched with a single docker run command. The code is Apache-2.0 licensed with about 3.8k GitHub stars, and adapters trained with PEFT or Ludwig work directly.