llama.cpp
Port of Meta's LLaMA model in C/C++ for efficient CPU inference
About
llama.cpp by Georgi Gerganov is a C and C++ inference engine for LLaMA-family and many other transformer language models, designed to run with minimal setup on a wide range of hardware including CPU-only laptops. It supports the GGUF quantized model format, multiple backends (CUDA, Metal, Vulkan, ROCm, BLAS), a server with an OpenAI-compatible API, and bindings for many languages. MIT licensed; the substrate for much of the local LLM ecosystem.
Reviews (0)
Leave a Review
No reviews yet. Be the first to review!
Details
- Category
- LLM Inference & Serving
- Price
- Free
- Platform
- Local/Desktop
- Difficulty
- Intermediate (3/5)
- License
- MIT
- Added
- Jan 29, 2026
Related Tools
Open-source ChatGPT alternative that runs 100% offline on your computer.
Fast LLM inference on consumer GPUs using neuron-aware sparse computation.
High-throughput LLM serving engine with PagedAttention
Easy-to-use local AI inference with built-in web UI and API.
High-performance LLM inference engine forked from vLLM with extra features.
Minimalist machine learning framework for Rust focused on performance and serverless inference.
Mentioned in
The Best Self-Hosted AI Stack for Small Teams in 2026
An opinionated reference architecture for self-hosted team AI in 2026: vLLM, Open WebUI, LiteLLM, Qdrant, and...
Billy C
Best Local LLM Setups by GPU Budget: 8GB, 16GB, 24GB, Multi-GPU
Exact model, quant, and engine picks for every VRAM tier in 2026, from an 8GB laptop GPU to a dual-3090...
Billy C
Running LLMs Without Nvidia in 2026: ROCm and Apple Silicon
The 2026 state of local LLMs off Nvidia: which runtimes actually work on AMD ROCm and Apple Silicon, real...
Billy C