KServe
Kubernetes platform for serving predictive models and LLMs with autoscaling, canaries and an OpenAI-style API.
About
Deploying a model with KServe means writing an InferenceService, a Kubernetes custom resource naming the model format and storage location, and letting the controller build the serving pods, routing and scaling around it. For generative workloads it runs vLLM or llm-d backends behind an OpenAI-compatible API, with model caching, KV cache offload to CPU or disk and request-driven autoscaling; for classic models it ships runtimes for PyTorch, TensorFlow, XGBoost, scikit-learn and ONNX. Optional transformer and explainer steps wrap the predictor, InferenceGraph chains models into pipelines or ensembles, and canary rollouts shift traffic gradually. Installs come in three shapes: a lightweight raw Kubernetes mode, a Knative serverless mode adding scale-to-zero and canaries, and ModelMesh for packing many frequently changing models densely. You need Kubernetes 1.32+, kubectl and Helm; kind or minikube works for local trials. It doubles as Kubeflow's serving add-on and is a CNCF incubating project. Apache-2.0 with no commercial restrictions. Best for platform teams already on Kubernetes that serve many models.
Should you use KServe?
Pick it when
Pick KServe when you already operate Kubernetes and want models served as standard custom resources across frameworks, with scale-to-zero for idle models and canary rollouts for new versions, managed like the rest of the cluster.
Look elsewhere when
Skip it if you do not already run Kubernetes; the operational floor is high for a handful of models, where BentoML or a single Triton or vLLM server is simpler. For the full ML lifecycle, Kubeflow bundles KServe with more.
Alternatives to KServe
- Seldon Core
Similar Kubernetes serving with A/B experiments and pipelines, but releases since January 2024 use the Business Source License, which needs a paid license for production.
- BentoML
No cluster needed to start: builds Docker images you can run anywhere, though autoscaling is yours to set up or comes from BentoCloud.
- Ray Serve
Composes several models and business logic in one Python application, where KServe favors declarative per-model resources.
- Triton Inference Server
A single high-throughput server with dynamic batching and ensembles on NVIDIA GPUs, without the Kubernetes control plane KServe adds.
Reviews (0)
Leave a Review
No reviews yet. Be the first to review!
Details
- Category
- AI Deployment & MLOps
- Price
- Free
- Platform
- Local/Desktop
- Difficulty
- Advanced (4/5)
- License
- Apache-2.0
- Added
- Apr 3, 2026