NVIDIA Dynamo
Distributed inference framework with disaggregated prefill and decode for datacenter LLM serving.
About
NVIDIA announced Dynamo at GTC 2025 as the open-source, datacenter-scale layer that orchestrates multi-node LLM serving without replacing the underlying engine, coordinating vLLM, SGLang, and TensorRT-LLM workers from one control plane. Its core techniques are disaggregated serving, which splits prefill and decode onto independently scalable GPU pools, KV-aware routing that sends requests to workers with the best cache overlap, and KV cache offloading tiered across GPU memory, CPU memory, SSD, and remote storage. ModelExpress streams weights GPU to GPU for cold starts the project reports as 7x faster, and an SLA-driven planner autoscales against latency and cost targets. The stack is written in Rust for performance with Python for extensibility, installs from PyPI as ai-dynamo with per-engine extras, and ships prebuilt containers on NGC plus beta zero-config Kubernetes deployment. Licensed Apache-2.0 with about 7.6k GitHub stars and documentation at docs.nvidia.com/dynamo, it targets teams serving large models across many nodes rather than single-GPU setups.
Should you use NVIDIA Dynamo?
Pick it when
You serve large models across many GPU nodes and want disaggregated prefill and decode, KV-aware routing, and tiered KV cache offloading on top of vLLM, SGLang, or TensorRT-LLM workers, ideally on an NVIDIA-heavy fleet.
Look elsewhere when
Skip it for single-GPU or single-node setups, where running vLLM or SGLang directly is simpler, since Dynamo only orchestrates engines. Kubernetes-first platform teams may prefer llm-d, a CNCF sandbox project with similar goals.
Alternatives to NVIDIA Dynamo
- llm-d
Similar cache-aware routing and prefill/decode disaggregation built on the Kubernetes Inference Gateway, but centered on vLLM rather than three engines.
- AIBrix
Kubernetes control plane for vLLM fleets that stands out on LoRA density and mixed GPU types, rather than multi-engine disaggregated serving.
- GPUStack
Easier multi-node serving across mixed vendors like AMD and Ascend from one docker run, aimed at pooling hardware more than datacenter-scale routing.
- SGLang
Has its own prefill-decode disaggregation and multi-node scaling, so for one engine a separate orchestration layer may be unnecessary.
Reviews (0)
Leave a Review
No reviews yet. Be the first to review!
Details
- Category
- LLM Inference & Serving
- Price
- Free
- Platform
- Local/Desktop
- Difficulty
- Advanced (4/5)
- License
- Apache-2.0
- Added
- Jul 29, 2026