WebLLM

Everything WebLLM does happens client-side: the engine downloads model weights into browser cache and runs LLM inference through WebGPU, so a chat application needs no inference server at all. The API is deliberately OpenAI-shaped, with streaming chat completions and JSON-schema structured generation, letting existing OpenAI client code retarget to in-browser models. It ships prebuilt support for Llama 3, Phi 3, Gemma, Mistral, and Qwen model families through the MLC LLM ecosystem, and custom models can be compiled with MLC's workflow; the project is the official companion to MLC LLM. Integration options cover dedicated web workers and service workers to keep the UI responsive, Chrome extensions, and multiple cache backends including the Cache API, IndexedDB, and OPFS. Installation is npm install @mlc-ai/web-llm or a CDN import. Apache-2.0 licensed with about 18.5k GitHub stars and a paper at arXiv 2412.15803, it is the standard way to put a genuinely private, offline-capable LLM inside a web page, provided the visitor's browser and GPU support WebGPU.