AIBrix

Born inside ByteDance and donated to the vLLM project in 2025, AIBrix is a cloud-native control plane that turns a Kubernetes cluster into scalable LLM serving infrastructure around the vLLM engine. The stack, written mostly in Go, adds an LLM-aware gateway that routes requests intelligently, an application-tailored autoscaler, high-density LoRA management for serving many adapters cheaply, a distributed KV cache for cross-engine reuse, heterogeneous GPU serving that mixes card types to cut cost, and GPU hardware failure detection, with a unified runtime sidecar handling metrics and model management. Deployment is Kubernetes-native: install the control plane with kubectl or kustomize manifests, then declare models as custom resources. The project is Apache-2.0 licensed, shipped v0.7.0 in June 2026, has passed 5,000 GitHub stars, and keeps its documentation on Read the Docs. Platform teams running vLLM fleets at scale use it as the orchestration layer that plain Kubernetes primitives do not provide.