How to Run AI Models Without a GPU in 2026: What Actually Works

If you want to run AI models without a GPU, the honest answer in 2026 is mostly yes for text, speech, embeddings and OCR, and still a firm no for video and large image models. This guide is for developers and indie makers on ordinary laptops, mini PCs and CPU-only servers who want realistic expectations: which tools to install, how much RAM each workload needs, and the exact commands. Every speed figure below comes from project documentation or a published benchmark, with the hardware named.

The short answer: what runs on CPU, task by task

The deciding question is how many weights a model has to stream per output step, and how many steps the job takes. Small, quantized models that emit text or audio pass that test. Diffusion models that push billions of parameters through dozens of denoising steps fail it.

The sections below explain each row, starting with the number that predicts LLM speed.

Why memory bandwidth matters more than core count

Generating one token means reading every active weight from memory once. Text generation on a CPU is therefore bound by memory bandwidth, and that gives you a ceiling you can compute: tokens per second cannot exceed bandwidth divided by the bytes read per token.

Take a typical desktop: dual-channel DDR5-5600 performs 5,600 million transfers per second, 8 bytes each, across two channels: about 89.6 GB/s. A Llama 3.1 8B file measures 14.96 GiB in BF16, 7.95 GiB at Q8_0 and 4.35 GiB at Q4_0, according to the llama-bench tables in a published ik_llama.cpp comparison. At Q4_0 that is about 4.7 GB per token, so the ceiling is roughly 19 tokens per second, and real throughput lands below it. In BF16 the same machine tops out near 6.

Two consequences follow:

• Quantization is not optional on CPU. Halving the bytes roughly doubles the ceiling.
• Memory channels beat cores. In llama.cpp's CPU performance discussion, one user's 96-core EPYC 9654 ran a 34B Q4_0 model at 1.5 tokens per second with a single 64 GB stick, and at 4 tokens per second after switching to two 32 GB sticks.

Prompt processing behaves differently. Your prompt is read in large batches, which leans on raw compute, so it scales with cores and SIMD width (AVX2, AVX-512, AMX). The same ik_llama.cpp comparison, run in December 2024, measured mainline llama.cpp on a 16-core Ryzen 7950X processing 8B prompts at about 153 tokens per second for Q4_0, 108 for Q4_K_S and 83 for Q6_K. Newer builds may do better, but paste a 4,000-token document into a setup like that and you could wait 25 to 50 seconds before the first output token. For CPU-only RAG, that delay hurts more than generation speed.

On threads, llama.cpp's performance guide warns that a -t value set too high oversaturates the CPU and recommends matching your physical core count. Its example table shows a 30B Q4_0 model generating 1.7 tokens per second on 7 physical cores with no GPU offload. The example is dated, but the bandwidth math still puts dense 30B-class models in low single digits.

How to run AI models without a GPU: llama.cpp and GGUF

llama.cpp (MIT) is the engine under most CPU-friendly LLM tools. It is plain C/C++ with AVX, AVX2, AVX512 and AMX code paths on x86, NEON on ARM, and 1.5-bit to 8-bit integer quantization stored in the GGUF format. The project tagged v0.6.0 on October 5, 2026, alongside its rolling numbered builds, which include CPU-only archives for Windows and Linux plus macOS binaries.

The quickest route is the new installer and the unified llama command:

# Linux and macOS (Windows: irm https://llama.app/install.ps1 | iex)
curl -LsSf https://llama.app/install.sh | sh

# chat with a small model pulled from Hugging Face
llama cli -hf ggml-org/Qwen3.5-0.8B-GGUF

# OpenAI-compatible API plus a built-in web UI
llama serve -hf ggml-org/Qwen3.5-0.8B-GGUF

The classic binaries still work. llama-server listens on 127.0.0.1:8080 by default, and these are the flags that matter on CPU:

llama-server -hf ggml-org/Qwen3.5-0.8B-GGUF -t 8 -c 8192 --cache-type-k q8_0 --cache-type-v q8_0

Set -t to your physical core count, -c to the context you actually need, and quantize the KV cache to q8_0 to keep memory in check as context grows.

For quantization, 4-bit K-quants such as Q4_K_M are the sensible CPU default: quality holds up and the bandwidth ceiling stays tolerable. Move to Q5 or Q6 only with RAM and bandwidth to spare, and treat 2-bit and 3-bit files as a last resort for fitting a model into memory.

Mixture-of-experts models change the math. Gemma 4 26B has 25.2B total parameters but only 3.8B active per token, so each token streams roughly what a 4B dense model would, while the whole 16 to 19 GB file still has to sit in RAM. On a 32 GB machine, a small-active MoE can be a better CPU choice than a dense 14B.

If you would rather not manage flags, several apps wrap llama.cpp. LM Studio adds a GUI but requires AVX2 on x64, recommends 16 GB of RAM, and does not support Intel Macs. llamafile (Apache 2.0, maintained by Mozilla.ai, version 0.10.6 at the time of writing) packs a model and runtime into one executable that runs on most operating systems and CPU architectures, and it bundles whisperfile for speech. KoboldCpp offers a koboldcpp-nocuda.exe build with CPU and Vulkan support for machines without NVIDIA hardware.

Ollama and the small models worth pulling

Ollama (MIT, v0.35.1 released September 29, 2026) wraps the same ideas in a model library and a REST API on localhost:11434. With no usable GPU it loads models into system memory, and it tells you so:

curl -fsSL https://ollama.com/install.sh | sh
ollama run qwen3.5:4b
ollama ps   # PROCESSOR column reads 100% CPU when the model is in system RAM

Without 24 GB or more of VRAM, Ollama's default context window is 4,096 tokens. Raise it with OLLAMA_CONTEXT_LENGTH=8192 ollama serve, and set OLLAMA_KV_CACHE_TYPE=q8_0 to cut KV cache memory to about half of f16. Cache quantization needs Flash Attention, which Ollama turns on automatically where the backend supports it.

The download size is your floor for RAM. Add the KV cache for your context length and leave headroom for the OS and other apps. Sizes below come from the Ollama library listings; the last column is a practical sizing suggestion, not a vendor requirement.

Other good small options include Microsoft's Phi-4-mini-instruct (3.8B, MIT) and Hugging Face's SmolLM3-3B (Apache 2.0). Models of 1B to 4B suit classification, extraction, routing and short answers; step up to 9B or a small-active MoE when the task needs longer reasoning.

Speech on CPU: whisper.cpp, faster-whisper, Piper and Kokoro

Speech to text

whisper.cpp (MIT, v1.9.5 released October 6, 2026) is the lowest-friction way to transcribe on a CPU, and its memory needs are modest. Per its README, tiny needs about 273 MB of RAM, base about 388 MB, small about 852 MB, medium about 2.1 GB and large about 3.9 GB.

git clone https://github.com/ggml-org/whisper.cpp && cd whisper.cpp
sh ./models/download-ggml-model.sh base.en
cmake -B build && cmake --build build -j --config Release
ffmpeg -i meeting.mp3 -ar 16000 -ac 1 -c:a pcm_s16le meeting.wav
./build/bin/whisper-cli -m models/ggml-base.en.bin -f meeting.wav -t 8

The ffmpeg step matters: whisper-cli currently accepts only 16-bit WAV input.

faster-whisper (MIT, 1.2.1) reimplements Whisper on CTranslate2 and publishes a clearly specified CPU benchmark: 13 minutes of audio, small model, Intel Core i7-12700K, 8 threads. openai/whisper took 6m58s in fp32. whisper.cpp took 2m05s using 1,049 MB, or 1m45s with its OpenVINO build. faster-whisper in int8 took 1m42s using 1,477 MB, and with batch_size=8 it finished in 51 seconds using 3,608 MB, roughly 15 times faster than real time.

from faster_whisper import WhisperModel, BatchedInferencePipeline

model = WhisperModel('small', device='cpu', compute_type='int8')
batched = BatchedInferencePipeline(model=model)
segments, info = batched.transcribe('meeting.mp3', batch_size=8)
for s in segments:
    print(f'[{s.start:.1f}s -> {s.end:.1f}s] {s.text}')

The whisper.cpp vs faster-whisper comparison goes deeper on accuracy trade-offs. For very small devices, Vosk's offline models are about 50 MB, cover 20+ languages and scale down to a Raspberry Pi or an Android phone.

Text to speech

Piper is the fast, dependable option. It now lives at OHF-Voice/piper1-gpl under the Open Home Foundation, and version 1.8.0 on PyPI is licensed GPL-3.0-or-later, which matters if you plan to ship it inside a closed product. It embeds espeak-ng for phonemization, and both Home Assistant and the NVDA screen reader use it.

pip install piper-tts
python3 -m piper.download_voices en_US-lessac-medium
python3 -m piper -m en_US-lessac-medium -f test.wav -- 'This is a test.'

The CLI reloads the voice on every call, so the docs recommend Piper's web server for repeated use.

Kokoro is the quality step up: an 82-million-parameter model with Apache-licensed weights, installed with pip install kokoro (0.9.4) plus espeak-ng for out-of-dictionary words and some languages. It outputs 24 kHz audio. To skip PyTorch, kokoro-onnx (0.6.1) runs the model on ONNX Runtime; its README lists the model at about 300 MB, or about 80 MB quantized, with near real-time speed on an M1 Mac.

from kokoro import KPipeline
import soundfile as sf

pipeline = KPipeline(lang_code='a')  # American English
for i, (_, _, audio) in enumerate(pipeline('Rendered on a laptop CPU.', voice='af_heart')):
    sf.write(f'line_{i}.wav', audio, 24000)

Voice cloning and LLM-based TTS models generate audio tokens through a language model backbone, so they inherit the slow-generation problem described above. The open-weight TTS roundup covers those models and their hardware needs.

Embeddings and OCR: the quiet CPU workhorses

Embedding models run one forward pass per input with no token-by-token generation, which suits CPUs well.

• all-MiniLM-L6-v2 has 22.7 million parameters, outputs 384-dimension vectors and is Apache 2.0. It remains the pick when throughput matters more than retrieval quality, and sentence-transformers (6.1.0) runs it on CPU out of the box.
• EmbeddingGemma has 308 million parameters, runs in under 200 MB of RAM with quantization according to Google, supports Matryoshka output from 768 down to 128 dimensions and has a 2K-token context. It ships under the Gemma license, and ollama pull embeddinggemma fetches a 622 MB build.
• Qwen3-Embedding-0.6B (Apache 2.0) offers a 32K context and up to 1,024 dimensions, which helps when chunks are long.

FastEmbed (Apache 2.0, 0.8.1) is the lightest way to run embedders in Python: it uses ONNX Runtime, needs no GPU and avoids pulling in PyTorch.

from fastembed import TextEmbedding

model = TextEmbedding()  # downloads the default model on first use
vectors = list(model.embed(['CPU inference is fine for embeddings.']))

OCR was CPU software long before the GPU era, and mostly still is. Tesseract 5.5.3 (July 2026, Apache 2.0) uses an LSTM line recognizer and reads more than 100 languages. OCRmyPDF (MPL-2.0) wraps it to add a searchable text layer to scanned PDFs. PaddleOCR 3.7.0 (Apache 2.0, June 2026) introduced PP-OCRv6 in tiny (1.5M), small (7.7M) and medium (34.5M parameter) tiers, with one model covering 50 languages, and the project claims a 5.2x end-to-end CPU speedup using OpenVINO. It can also run on ONNX Runtime via --engine onnxruntime; without that flag it uses the PaddlePaddle framework, and the docs advise one inference engine per environment.

tesseract scan.png out -l eng          # writes out.txt
ocrmypdf scanned.pdf searchable.pdf    # Tesseract under the hood
python -m pip install paddleocr
python -m pip install paddlepaddle==3.2.0 -i https://www.paddlepaddle.org.cn/packages/stable/cpu/
paddleocr ocr -i scan.png --device cpu --save_path ./output

For full PDF-to-Markdown conversion, MinerU's basic tier runs ONNX models (about a 0.8 GB download) on CPU with a 2 GB RAM minimum, and its standard tier adds llama.cpp with an 8 GB minimum, where CPU works and Vulkan is recommended. Parsers built on vision-language models are where CPU starts to drag.

ONNX Runtime and OpenVINO: more speed from the same x86 box

ONNX Runtime (MIT, 1.30.0 released September 10, 2026) is the dependency behind FastEmbed, kokoro-onnx and Piper. Export a model to ONNX once and the same file runs on Windows, Linux and macOS with graph optimizations applied, using hardware accelerators where they exist and the CPU everywhere else.

OpenVINO (Apache 2.0, 2026.4.1 released October 1, 2026) is Intel's inference toolkit and the strongest option on Intel machines. It targets x86 and ARM CPUs, Intel integrated and discrete GPUs, and Intel NPUs, so an Intel laptop with an integrated GPU or NPU can move work off the CPU cores without a discrete card. OpenVINO GenAI provides LLM, Whisper and image pipelines, and Optimum Intel handles conversion:

pip install -U optimum-intel openvino-genai
optimum-cli export openvino --model TinyLlama/TinyLlama-1.1B-Chat-v1.0 --weight-format int4 --trust-remote-code TinyLlama_ov

import openvino_genai as ov_genai

pipe = ov_genai.LLMPipeline('TinyLlama_ov', 'CPU')  # or 'GPU' with no other code change
print(pipe.generate('What is OpenVINO?', max_new_tokens=100))

llama.cpp is adding an OpenVINO backend (marked in progress), and its numbered builds already include OpenVINO archives for Linux and Windows.

Apple Silicon: the unified memory middle ground

Apple Silicon sidesteps the question: its integrated GPU reads the same unified memory as the CPU, at higher bandwidth than a typical PC. llama.cpp's Apple Silicon benchmark thread (LLaMA 7B at Q4_0, text generation) shows speed closely tracking bandwidth: an M1 at 68 GB/s produces about 14.2 tokens per second, an M4 at 120 GB/s about 24.1, a 10-core-GPU M5 at 154 GB/s about 31.9, and a 40-core M4 Max at 546 GB/s about 83.1. Prompt processing on the base M4 runs at about 221 tokens per second, ahead of the 16-core desktop figure above.

Apple's March 2026 MacBook Pro update offers the M5 Max with up to 128 GB of unified memory and up to 614 GB/s of bandwidth. The catch: by default macOS lets Metal use only about two thirds to three quarters of unified memory. Raise the cap with sudo sysctl iogpu.wired_limit_mb=<megabytes>; the setting resets at reboot and should stay well below total RAM.

MLX is Apple's array framework, and mlx-lm (0.32.0, October 2026) is the native alternative to llama.cpp on Macs. It also supports LoRA fine-tuning, including on quantized models, which makes small-model fine-tuning on a laptop realistic. Ollama's library now carries -mlx tags for models such as Gemma 4 and Qwen3.5.

pip install mlx-lm
mlx_lm.generate --prompt 'How tall is Mt Everest?'
mlx_lm.chat

The AMD and Apple Silicon guide covers hardware buying decisions in more depth.

What does not run acceptably without a GPU

Three categories still call for a GPU, owned or rented.

Video generation. Wan 2.2's README says its 14B text-to-video and image-to-video commands need a single GPU with at least 80 GB of VRAM, and its efficient TI2V-5B model needs at least 24 GB (an RTX 4090), with the project quoting under 9 minutes per 5-second 720P clip on a single consumer GPU. The smaller Wan 2.1 1.3B model needs 8.19 GB of VRAM and about 4 minutes per 5-second 480P clip on an RTX 4090. Those times come from a top consumer GPU; a CPU has a fraction of that parallel compute, so treat Wan, HunyuanVideo and their peers as GPU-only.

Large image models. FLUX.2-dev has about 32 billion parameters and Qwen-Image about 20 billion. Every image runs many denoising steps through the full network, the compute-bound work where CPUs fall furthest behind. stable-diffusion.cpp does run on CPU with AVX, AVX2 and AVX512, and it is the right tool if you insist: stay with SD 1.x, SD-Turbo or SDXL-Turbo class models at low step counts and expect to wait on every image.

Training. Fine-tuning runs forward and backward passes over large token counts, which is compute-bound again. Unsloth lists CPUs among its supported backends and MLX makes LoRA on small models workable on a Mac, but full fine-tunes and LoRA on 7B-plus models belong on a GPU.

When the answer is a GPU, the local LLM setups by GPU budget guide maps what each price tier buys.

Common questions

Can I run an LLM on a laptop with 8 GB of RAM? Yes, if you stay small. Qwen3.5 0.8B (about 1.2 GB) and 4B (3.3 to 4.0 GB) fit with room for a modest context, and llama.cpp or Ollama runs them on the CPU alone.

How many tokens per second should I expect on CPU? Divide memory bandwidth by model file size for a ceiling. Dual-channel DDR5 at about 90 GB/s with a 4.7 GB 8B Q4_0 file gives roughly 19 tokens per second at best, and real results land lower; MoE models with few active parameters beat what their total size suggests.

Is Whisper usable without a GPU? Yes. faster-whisper in int8 transcribed 13 minutes of audio in 1m42s with the small model on an i7-12700K, or in 51 seconds with batching.

Does Ollama work without a GPU? Yes. It loads the model into system memory, ollama ps shows 100% CPU, and the same models and API work, just more slowly.

Is a Mac better than a PC for local AI without a discrete GPU? For LLMs, usually. A base M4 gives its integrated GPU 120 GB/s of unified memory bandwidth, against roughly 90 GB/s for dual-channel DDR5, and Max chips go several times higher.