NVIDIA Dynamo

NVIDIA announced Dynamo at GTC 2025 as the open-source, datacenter-scale layer that orchestrates multi-node LLM serving without replacing the underlying engine, coordinating vLLM, SGLang, and TensorRT-LLM workers from one control plane. Its core techniques are disaggregated serving, which splits prefill and decode onto independently scalable GPU pools, KV-aware routing that sends requests to workers with the best cache overlap, and KV cache offloading tiered across GPU memory, CPU memory, SSD, and remote storage. ModelExpress streams weights GPU to GPU for cold starts the project reports as 7x faster, and an SLA-driven planner autoscales against latency and cost targets. The stack is written in Rust for performance with Python for extensibility, installs from PyPI as ai-dynamo with per-engine extras, and ships prebuilt containers on NGC plus beta zero-config Kubernetes deployment. Licensed Apache-2.0 with about 7.6k GitHub stars and documentation at docs.nvidia.com/dynamo, it targets teams serving large models across many nodes rather than single-GPU setups.