Xinference

One deployment of Xinference (Xorbits Inference) can serve language models, text embeddings, speech recognition, text-to-image, and other multimodal models behind a single OpenAI-compatible REST API that includes function calling support. Under the hood it delegates to pluggable backends such as vLLM, llama.cpp via the xllamacpp bindings, and TensorRT, picking engines per model, and it scales from a laptop launched with xinference-local up to distributed multi-node clusters that pool heterogeneous GPUs and CPUs. Interfaces include the REST API, a CLI, RPC, and a built-in web UI for launching and managing models from a large built-in catalog. Installation is pip install xinference[all], an official Docker image, or Helm charts for Kubernetes. The core is Apache-2.0 licensed with about 9.5k GitHub stars and has reached the v3.0.0 series, while a commercial enterprise edition funds development. Self-hosted RAG stacks commonly use it as the single model-serving layer behind frameworks that expect OpenAI-style endpoints, since one process covers chat, embedding, and rerank-style workloads.