OpenLLM
Runs open-source LLMs as OpenAI-compatible API endpoints with a single command and built-in chat UI.
About
A single openllm serve command turns an open-weight model into an OpenAI-compatible API endpoint with a web chat UI at /chat, which is the entire pitch of OpenLLM, BentoML's LLM serving tool. The CLI pulls from a curated model repository covering Llama 3 and 4, Mistral, Qwen 2.5, DeepSeek, Phi-4, and Gemma variants, runs them on vLLM-backed inference for production throughput, and exposes chat completions that work as drop-in replacements for OpenAI clients. Installation is pip install openllm, and an interactive openllm hello command walks through the available models. Because it is built on BentoML, the same service definition deploys to BentoCloud for managed autoscaling, though nothing requires the hosted platform and self-hosting on your own GPUs is the default path. The project is Apache-2.0 licensed with about 12.4k GitHub stars and stays actively maintained by the BentoML team. Developers reach for it when they want the shortest path from a Hugging Face model name to a production-shaped, OpenAI-compatible endpoint on their own hardware.
Reviews (0)
Leave a Review
No reviews yet. Be the first to review!
Details
- Category
- LLM Inference & Serving
- Price
- Free
- Platform
- Local/Desktop
- Difficulty
- Easy (2/5)
- License
- Apache-2.0
- Added
- Jul 29, 2026
Related Tools
Minimalist ML framework in Rust by Hugging Face for fast inference.
Open-source ChatGPT alternative that runs 100% offline on your computer.
Port of Meta's LLaMA model in C/C++ for efficient CPU inference
Fast LLM inference on consumer GPUs using neuron-aware sparse computation.
High-throughput LLM serving engine with PagedAttention
Minimalist machine learning framework for Rust focused on performance and serverless inference.