BentoML

BentoML turns model inference code into production services. Developers define a service in Python with type-hinted APIs, run it locally, then package the model, code, and dependencies into a standardized artifact called a Bento that builds into a Docker image for deployment anywhere. The framework is agnostic to ML framework and modality, serving LLMs, embeddings, image generation, audio, and computer vision workloads. Performance features include adaptive request batching, worker parallelization, model composition, multi-stage inference pipelines, and GPU support with CUDA detection, while built-in observability covers monitoring out of the box. Beyond self-managed Docker or Kubernetes deployments, the company behind it offers BentoCloud as a managed platform with autoscaling and distributed serving. BentoML is open source under the Apache 2.0 license and is used by ML platform and backend teams that need reliable, cost-efficient inference APIs and multi-model serving systems.