vLLM

Serving large language models efficiently on GPUs is the problem vLLM targets, and more than 2,000 contributors now maintain it. Its PagedAttention design stores the attention KV cache in fixed-size blocks, much like virtual memory pages, so many concurrent requests can share GPU memory with little waste; on top of that sit continuous batching, prefix caching, chunked prefill, speculative decoding and CUDA graph execution. Hundreds of Hugging Face architectures load directly, from dense and mixture-of-experts LLMs to multimodal, embedding and reward models, with quantized formats such as FP8, NVFP4, GPTQ, AWQ and GGUF. The vllm serve command exposes an OpenAI compatible API, plus Anthropic Messages and gRPC endpoints, with tool calling, structured output and multi-LoRA. Models can span GPUs and nodes via tensor, pipeline, data and expert parallelism. Install with uv or pip on Linux and Python 3.10 to 3.13; NVIDIA GPUs need compute capability 7.5 or newer, AMD, Intel, CPUs, TPUs and other accelerators work through backends and plugins, and Windows needs WSL. Apache-2.0 licensed, so commercial deployment is unrestricted.