Triton Inference Server

NVIDIA's Triton Inference Server lets one process host many models from different frameworks behind a shared HTTP/REST and gRPC interface that follows the KServe v2 protocol. Models sit in a versioned model repository, and pluggable backends run them: TensorRT, TensorRT-LLM, vLLM, ONNX Runtime, PyTorch, OpenVINO, RAPIDS FIL, and a Python backend for custom logic. The scheduler adds dynamic batching, sequence batching with implicit state, several concurrent instances of a model across GPUs, and ensembles or business logic scripting to chain pre and post processing. Prometheus metrics expose GPU utilization, throughput and latency, and C and Java in-process APIs allow embedding at the edge. The usual route is the prebuilt tritonserver container from NGC with a mounted model repository; CPU-only and ARM builds exist, though NVIDIA GPUs gain the most. Companion repos supply Python, C++ and Java clients and perf_analyzer for load testing. The server is BSD-3-Clause, so commercial use is fine, with paid support sold through NVIDIA AI Enterprise. Aimed at MLOps teams serving mixed model types in production.