Ray Serve

Ray Serve handles online inference on top of the Ray distributed computing framework. It stays framework agnostic, serving PyTorch, TensorFlow, Keras, and scikit-learn models alongside arbitrary Python business logic, and its distinguishing feature is model composition: multiple models and processing steps combine into one application through a programmable Python API instead of static configuration files. Each component autoscales independently, deployments can claim fractional GPUs to pack models onto shared hardware, and LLM-oriented features include response streaming, dynamic request batching, and multi-node multi-GPU serving. HTTP handling integrates with FastAPI for request parsing and validation. The same code runs on a laptop, on Kubernetes, or on cloud and on-premise clusters. Distributed as part of the open source Ray project under the Apache 2.0 license, it is used by ML engineers and platform teams building scalable inference APIs without vendor lock-in.