llm-d
Kubernetes-native distributed inference stack combining vLLM with KV-cache-aware routing at scale.
About
Red Hat, Google Cloud, IBM Research, CoreWeave, and NVIDIA founded llm-d in 2025 to give Kubernetes a native way to serve large language models at scale, and the project entered the CNCF sandbox in March 2026. It layers three pieces: vLLM as the execution engine, the Kubernetes Inference Gateway for orchestration, and KV-cache-aware routing that balances requests by prefix-cache hits and worker load rather than round robin. On top of that come tiered KV cache offloading to CPU memory and disk, prefill/decode disaggregation for serving very large models, SLO-aware autoscaling, and OpenAI-compatible batch processing APIs. Installation uses Helm charts with published quickstart paths, and the v0.7 release from May 2026 brought predicted-latency scheduling to GA plus a kustomize-first deployment flow. Everything is Apache-2.0 licensed with about 3.9k GitHub stars. Platform engineers pick it over hand-rolled vLLM deployments when they need cache-aware load balancing and disaggregation without building the control plane themselves.
Should you use llm-d?
Pick it when
Pick llm-d when a platform team serves large models on Kubernetes and needs prefix-cache-aware load balancing, prefill/decode disaggregation, and SLO-based autoscaling around vLLM, under vendor-neutral CNCF governance.
Look elsewhere when
Skip it without an existing Kubernetes practice or for a handful of GPUs; vLLM alone or GPUStack is far simpler. It is a young project, and teams wanting TensorRT-LLM or SGLang workers too may prefer NVIDIA Dynamo.
Alternatives to llm-d
- AIBrix
Also Kubernetes plus vLLM, with stronger focus on high-density LoRA serving and mixed GPU types, but without CNCF governance.
- NVIDIA Dynamo
Coordinates vLLM, SGLang, and TensorRT-LLM rather than vLLM only, with similar disaggregation, but its zero-config Kubernetes deployment is still beta.
- LMCache
Just the KV cache tier, added to an existing vLLM or SGLang deployment, when you want cache reuse without adopting a full control plane.
- GPUStack
Pools GPUs across machines from a docker run with no Kubernetes skills, but it is a cluster manager, not a cache-aware routing layer.
Reviews (0)
Leave a Review
No reviews yet. Be the first to review!
Details
- Category
- LLM Inference & Serving
- Price
- Free
- Platform
- Local/Desktop
- Difficulty
- Advanced (4/5)
- License
- Apache-2.0
- Added
- Jul 29, 2026
Tags
Related Tools
Browse all LLM Inference & Serving toolsMentioned in
LLM Gateways in Production: LiteLLM vs Portkey vs Bifrost
A working comparison of LiteLLM, Portkey, and Bifrost on routing, caching, budgets, observability, and real...
Max P
Serving LLMs on Kubernetes: llm-d, AIBrix, and Dynamo
How llm-d, AIBrix, NVIDIA Dynamo, GPUStack, OpenLLM and Xinference actually differ on Kubernetes, and when a...
Billy C