LitServe

Lightning AI built LitServe as a thin serving layer on top of FastAPI for teams that want a custom inference server without adopting a heavyweight MLOps stack. A model gets wrapped in a LitAPI class with setup, decode, predict, and encode hooks; the framework then supplies the parts plain FastAPI lacks for ML traffic: dynamic request batching, token and byte streaming, multi-worker and multi-GPU autoscaling, and optional OpenAI-compatible endpoints, which Lightning benchmarks at more than 2x plain FastAPI throughput. Because the serving loop is arbitrary Python, the same pattern serves LLMs, vision models, audio, RAG pipelines, agents, and classical ML from PyTorch, JAX, or TensorFlow, and roughly 100 community templates cover common models. Installation is pip install litserve under Apache 2.0 with no restriction on commercial use; servers run on a laptop, an on-prem GPU box, or any cloud, and a one-command deploy to Lightning's managed cloud exists for teams that would rather not run infrastructure.