WebLLM
In-browser LLM inference engine that runs models client-side over WebGPU with an OpenAI-style API.
About
Everything WebLLM does happens client-side: the engine downloads model weights into browser cache and runs LLM inference through WebGPU, so a chat application needs no inference server at all. The API is deliberately OpenAI-shaped, with streaming chat completions and JSON-schema structured generation, letting existing OpenAI client code retarget to in-browser models. It ships prebuilt support for Llama 3, Phi 3, Gemma, Mistral, and Qwen model families through the MLC LLM ecosystem, and custom models can be compiled with MLC's workflow; the project is the official companion to MLC LLM. Integration options cover dedicated web workers and service workers to keep the UI responsive, Chrome extensions, and multiple cache backends including the Cache API, IndexedDB, and OPFS. Installation is npm install @mlc-ai/web-llm or a CDN import. Apache-2.0 licensed with about 18.5k GitHub stars and a paper at arXiv 2412.15803, it is the standard way to put a genuinely private, offline-capable LLM inside a web page, provided the visitor's browser and GPU support WebGPU.
Should you use WebLLM?
Pick it when
You want a private, offline-capable chat LLM running entirely inside a web page or Chrome extension, with OpenAI-style streaming and JSON-schema output, and can accept that visitors need a WebGPU-capable browser and GPU.
Look elsewhere when
Skip it when your audience includes browsers or devices without WebGPU, or when models are too large to download per visitor; a server engine like vLLM or Ollama fits then. For embeddings, speech, or vision, Transformers.js covers more.
Alternatives to WebLLM
- Transformers.js
Broader task coverage, including classification, speech, vision, and embeddings, but its LLM API is less chat-focused and OpenAI-shaped.
- MLC LLM
The same compiler stack extended to native iOS, Android, and server targets, for when the app is not browser-only.
- Ollama
Runs a local server on the user's machine instead, so no per-visitor model download, but every user must install it first.
Reviews (0)
Leave a Review
No reviews yet. Be the first to review!
Details
- Category
- LLM Inference & Serving
- Price
- Free
- Platform
- Local/Desktop
- Difficulty
- Intermediate (3/5)
- License
- Apache-2.0
- Added
- Jul 29, 2026