SGLang

SGLang is a serving framework for large language models and vision-language models, developed under LMSYS, focused on low-latency, high-throughput inference. Its runtime combines RadixAttention prefix caching, continuous batching, paged attention, a zero-overhead CPU scheduler, prefill-decode disaggregation, and speculative decoding to scale from a single GPU to large multi-node clusters. Structured output is a distinguishing feature: constrained decoding driven by a compressed finite state machine produces JSON and regex-constrained text substantially faster than conventional approaches. The framework supports model families including Llama, Qwen, and DeepSeek along with embedding and diffusion models, exposes an OpenAI-compatible API, and offers FP8, FP4, and INT4 quantization. Hardware support spans NVIDIA and AMD GPUs, Intel Xeon CPUs, Google TPUs, and Ascend NPUs. The project reports production deployments across hundreds of thousands of GPUs at companies including xAI and major cloud providers, and the code is open source under the Apache 2.0 license.