TabbyAPI

Built by kingbri, Splice86, and turboderp, this FastAPI server puts an OpenAI-style HTTP API in front of the ExLlamaV3 inference library, so OpenAI-compatible clients can use models on your own GPU. It loads EXL3 quantized weights, which the authors recommend, or plain FP16 and BF16 checkpoints; the older ExLlamaV2 backend, and with it EXL2 and GPTQ support, was removed in mid-2026. Features include continuous batching with paged attention on Ampere and newer NVIDIA cards, speculative decoding with a draft model, tool calling, JSON schema, regex, and EBNF constrained output, Jinja2 chat templates, model swapping over the API, Hugging Face downloads, an embeddings endpoint through infinity, and AI Horde participation. You need an NVIDIA GPU (Turing or newer; Volta is rejected) and Python 3.10 up to 3.14, or the published CUDA Docker image; ROCm support arrives with the next ExLlamaV3 release. The authors call it a rolling-release hobby project, not a production server. Code is AGPL-3.0, so modified versions offered as a network service must share source. Suited to enthusiasts fitting large quantized models onto consumer GPUs.