llm-d

Red Hat, Google Cloud, IBM Research, CoreWeave, and NVIDIA founded llm-d in 2025 to give Kubernetes a native way to serve large language models at scale, and the project entered the CNCF sandbox in March 2026. It layers three pieces: vLLM as the execution engine, the Kubernetes Inference Gateway for orchestration, and KV-cache-aware routing that balances requests by prefix-cache hits and worker load rather than round robin. On top of that come tiered KV cache offloading to CPU memory and disk, prefill/decode disaggregation for serving very large models, SLO-aware autoscaling, and OpenAI-compatible batch processing APIs. Installation uses Helm charts with published quickstart paths, and the v0.7 release from May 2026 brought predicted-latency scheduling to GA plus a kustomize-first deployment flow. Everything is Apache-2.0 licensed with about 3.9k GitHub stars. Platform engineers pick it over hand-rolled vLLM deployments when they need cache-aware load balancing and disaggregation without building the control plane themselves.