llama.cpp

Running an open-weight language or vision model on your own hardware without a Python stack usually leads here. The engine, plain C and C++ built over the ggml tensor library, loads GGUF files quantized from about 1.5 to 8 bits per weight and runs them on CPUs with AVX, AMX, ARM NEON or RISC-V vector extensions, or on GPUs through CUDA, HIP, Metal, Vulkan, SYCL, OpenCL, MUSA, CANN and WebGPU backends, offloading part of a model to the GPU when it does not fit in VRAM. It ships a CLI that pulls models straight from Hugging Face, plus llama-server, an HTTP server with OpenAI-style chat, responses and embeddings routes, Anthropic Messages compatibility, reranking, continuous batching, function calling, speculative decoding, GBNF grammar constraints and a browser chat UI. Install with a one-line script, winget, Docker, prebuilt release binaries or a CMake build, or use llama.app, a small desktop wrapper the team built with Hugging Face. MIT licensed, so it can be embedded in commercial products. Anyone serving models on a laptop, an edge board or a lean cloud container should know it.