mistral.rs

Despite the name, mistral.rs runs far more than Mistral models: it is a cross-platform inference engine written in Rust that covers text, vision, audio, and video input plus speech generation, image generation, and embeddings. Quantization is a core strength, with in-situ quantization that converts Hugging Face models on the fly, GGUF support from 2 to 8 bits, GPTQ, AWQ, HQQ, and FP8, and per-layer topology control that lets each layer use a different format. The server exposes OpenAI-compatible /v1 endpoints alongside an Anthropic-style Messages API, a built-in web UI, and a Prometheus metrics endpoint, with Rust and Python APIs for embedding it in applications. It runs on CUDA with FlashAttention V2 and V3, Apple Metal, and plain CPU, and supports multi-GPU distributed inference. Prebuilt binaries install through shell or PowerShell one-liners on Linux, macOS, and Windows, falling back to a source build. Maintained by Eric Buehler under the MIT license, it has around 7.5k GitHub stars as a popular Rust-native alternative to llama.cpp.