llama-cpp-python

Python bindings for Georgi Gerganov's llama.cpp turn that C++ inference library into something installable with pip and usable from ordinary Python code. llama-cpp-python exposes three layers: low-level ctypes bindings to the C API, a high-level interface for text and chat completion, and an OpenAI-compatible web server that works as a drop-in replacement for hosted APIs. It supports function and tool calling in the OpenAI format, multimodal input through models such as LLaVA and Moondream, JSON schema constrained output, speculative decoding with draft models, and embedding generation. Builds compile llama.cpp from source, with CMake flags enabling CUDA, Metal, ROCm, Vulkan, SYCL, or OpenBLAS backends, and models can be pulled directly from the Hugging Face Hub. The library integrates with LangChain and LlamaIndex and is released under the MIT license, making it a common foundation for local assistants and privacy-sensitive deployments.