Best Local LLM Setups by GPU Budget: 8GB, 16GB, 24GB, Multi-GPU

Four VRAM budgets, four working stacks. This guide gives concrete model, quantization, and inference engine picks for 8GB, 16GB, 24GB, and multi-GPU local LLM setups as of August 2026, written for developers who want to pull weights and get to work rather than tour a leaderboard. Every pick is a specific artifact you can download today, with honest notes about where it falls over.

How VRAM math works in 2026

Three shifts changed the arithmetic since the 70B-or-bust days, and they decide every pick below.

First, mixture-of-experts went mainstream at consumer scale. Models like Qwen3.6-35B-A3B (35B total, roughly 3B active per token), GLM-4.7-Flash (30B-A3B), and OpenAI's gpt-oss 20B (21B total, 3.6B active) split the bill: total parameters set your memory cost, active parameters set your speed. A 35B MoE at Q4 fits where a 35B dense model would choke, and decodes several times faster.

Second, hybrid attention shrank the KV cache. The Qwen3.6 line runs a 3:1 ratio of linear attention to full attention layers, which is why a 27B dense model can hold 64K context inside 24GB while Gemma 4 31B, whose interleaved sliding-window attention still carries a far heavier cache, runs out of memory much earlier on the same card. In 2026, context headroom is an architecture property, not just a VRAM number.

Third, expert offload became a first-class feature. llama.cpp can pin attention and shared layers on the GPU while pushing MoE expert weights to system RAM with the --n-cpu-moe flag. That is how a 24GB card runs models whose weights alone exceed 24GB, at a speed penalty you can tune layer by layer.

The baseline rule still holds: Q4_K_M weights cost roughly 0.6GB per billion parameters, and you should budget 1 to 2GB extra for KV cache and runtime at short context, more as context grows.

Pick the engine before the model

The engine question resolves faster than people think, because the categories barely overlap.

For one user on one machine, the llama.cpp family is the default: GGUF quants, CPU and GPU hybrid offload, and it runs on CUDA, Metal, ROCm, Vulkan, and bare CPUs. Ollama wraps it with model management and one-line pulls; LM Studio puts a GUI and an OpenAI-compatible server on top and adds MLX on Apple Silicon; KoboldCpp and llamafile serve narrower niches. Pair any of them with Open WebUI if you want a chat front end.

For serving multiple users or agent fleets, vLLM is the standard: PagedAttention, continuous batching, tensor parallelism, and first-class AWQ, GPTQ, and FP8 support. SGLang competes directly and its RadixAttention prefix caching wins when many requests share long prompt prefixes, which describes most agent workloads. TensorRT-LLM is the speed ceiling on NVIDIA, reported at 10 to 20 percent more throughput than vLLM, but vLLM delivers about 85 percent of that performance with far less operational pain, which is the right trade for almost everyone.

For a single user with two to four consumer NVIDIA cards, the ExLlama line is the enthusiast frontier. ExLlamaV3 and its EXL3 quant format (successor to ExLlamaV2 and EXL2) target maximum single-stream speed on exactly this hardware, served through TabbyAPI. And for giant MoE models that mostly live in system RAM with one GPU assisting, KTransformers is the specialist tool.

Rule of thumb: weird or modest hardware, llama.cpp family. Team endpoint, vLLM or SGLang. Multi-GPU hot rod for one person, ExLlamaV3 via TabbyAPI.

8GB: no longer a toy tier

The pick: Qwen3.5-9B at Q4_K_M, on Ollama or llama.cpp. Alibaba shipped the Qwen3.5 small series (0.8B, 2B, 4B, and 9B dense models) in early 2026, under Apache 2.0, with native text and image input and a hybrid Gated DeltaNet attention design. The 9B lands around 5.7GB at Q4_K_M, which leaves genuine context headroom on an 8GB card, and it beats the entire 2024-2025 crop of 7B and 8B models on essentially every axis.

ollama run qwen3.5:9b
# or, llama.cpp with everything on GPU:
llama-server -m qwen3.5-9b-Q4_K_M.gguf -ngl 99 --flash-attn on

Alternatives by job:

• Math and reasoning: DeepSeek-R1-0528-Qwen3-8B, the 8B distill from DeepSeek-R1, MIT licensed, about 5.2GB at Q4_K_M. It burns tokens on visible chain of thought, so it is slower per answer but markedly better at hard problems.
• Code autocomplete: Qwen2.5-Coder 7B at Q4_K_M is about 4.7GB and remains the practical choice for local tab-completion backends where latency matters more than depth.
• Vision and documents: GLM-4.6V-Flash (about 10B, MIT) squeezes in at roughly 6GB quantized, with less room for context.
• 8GB of plain RAM, no GPU: drop to Qwen3.5-4B (about 3GB at Q4), Phi-4 Mini at about 2.5GB, or SmolLM3 at about 2GB. Google's Gemma 4 E-series QAT builds (the E2B is built to run in a few gigabytes) are the multimodal option here, a clear step past Gemma 3 4B.

Honest limits: 8GB setups now write competent code snippets and summarize well, but agentic coding loops and 32K-plus contexts still thrash. Do not fight it; this tier is for chat, drafting, extraction, and single-file code help.

16GB: the new practical baseline

The pick: gpt-oss-20b. OpenAI's August 2025 open-weights MoE (gpt-oss, Apache 2.0) ships natively in MXFP4 4-bit, so there is no quantization decision to make and nothing lost by making one. It occupies 12 to 14GB, and because only 3.6B parameters activate per token it is fast even on weak silicon: independent testing on a bandwidth-starved NVIDIA T4 has measured throughput holding essentially flat from 1K all the way out to 60K context, collapsing only near the context ceiling when the cache spills. On a modern 16GB card like a 5060 Ti or 4080 you get the same flat profile at higher speed. Structured tool calling is its strong suit; prose style is its weak one.

The coding pick: Devstral Small 2. Mistral's 24B agentic coder (Devstral, released December 2025, Apache 2.0, 256K context, image input) scores a reported 68.0 percent on SWE-bench Verified, the best figure of anything that fits this tier, and within five points of its own 123B sibling. At Q4_K_M it needs 14 to 15GB, which on a 16GB card means aggressive context trimming or an IQ4_XS quant; it truly relaxes at 24GB. If your local model exists to drive Cline, OpenHands, or another agent harness, this is the one to squeeze in.

Worth knowing at this tier: Gemma 4 12B (about 7GB at Q4, 256K context, multimodal) is the low-footprint all-rounder; Qwen3 14B remains a fine dense generalist at about 9GB; and Qwen3.6-35B-A3B becomes reachable via expert offload, trading speed for a much smarter model:

# 35B MoE on a 16GB card: park some experts in system RAM
llama-server -m qwen3.6-35b-a3b-Q4_K_M.gguf -ngl 99 --flash-attn on --n-cpu-moe 16

24GB: one card, serious work

This is the tier where local stops feeling like a compromise, and the 2026 strategy is clear: run modern 20B to 35B models that fit cleanly, not last era's 70B at starvation quants.

Best overall: Qwen3.6-27B. Released in April 2026, Apache 2.0, dense, about 16GB at Q4_K_M with a 262K context window. Its hybrid attention keeps the KV cache small enough that the full 64K context runs entirely on a 24GB GPU, and its agentic coding results (a reported 77.2 on SWE-bench Verified) embarrassed several closed models at launch. It is not the fastest dense 27B you have ever run, and its visible reasoning mode spends tokens freely, but as a single default on this card it currently has no serious rival.

Best throughput: Qwen3.6-35B-A3B. Same family, also April 2026, MoE with about 3B active. On identical hardware it decodes several times faster than the dense 27B, and with a dozen experts parked in system RAM via --n-cpu-moe it still holds comfortable interactive speeds at 64K context with VRAM to spare. When agents need many fast turns rather than one deep one, run this.

Multimodal: Gemma 4. The 31B posts standout science and vision numbers (a reported 84.3 on GPQA Diamond) but pays for it with a KV cache that forces context down on a 24GB card. The MoE sibling, Gemma 4 26B-A4B, is the pragmatic choice: MoE-class decode speed that holds up well out to 64K context.

Tool-use specialist: GLM-4.7-Flash. Z.ai's MIT-licensed 30B-A3B posts strong reported tool-calling numbers (79.5 on TAU2-Bench) and is a favorite agent backbone in the GLM line, but its Q4_K_M weights alone approach 20GB, so on 24GB you take a smaller quant or offload a few experts.

Still respectable: Mistral Small 3.2 24B (about 14GB at Q4) as a low-latency assistant, and the MIT-licensed DeepSeek-R1-Distill-Qwen-32B at 18 to 20GB if you want 2025's reasoning distill; both have been outclassed by the spring 2026 wave but lose gracefully.

On engines: Ollama or llama.cpp is fine for solo use; a 4090 owner chasing single-stream speed should try an EXL3 quant under TabbyAPI; and if the card serves a small team, an AWQ build under vLLM will batch far better than GGUF anywhere.

48GB and beyond: multi-GPU without the mythology

Two used RTX 3090s remain the canonical budget path to 48GB, and the folklore around them deserves some numbers. Over PCIe, tensor-parallel inference of a 70B model scales at roughly 0.7x of ideal, so two cards buy you about 1.4 cards of throughput. NVLink helps engines that do real tensor parallelism, with reports of 40 to 60 percent faster inference than PCIe on paired 3090s, but llama.cpp does not use NVLink at all; its --tensor-split layer splitting just needs both cards reachable. Know which camp your engine is in before buying a bridge.

What 48GB actually buys in 2026 is more interesting than what it bought in 2024:

• The old move: Llama 3.3 70B at Q4_K_M is about 42.5GB and runs at 18 to 22 tokens per second on dual 3090s, with only 4 to 8K of comfortable context. It is still the strongest open dense generalist, but that is a eulogy as much as a title.
• The 2026 move: run a modern 27B to 35B at Q8 with enormous context instead. Qwen3.6-27B at Q8 with long context beats Llama 3.3 70B Q4 on most agentic tasks in recent 48GB shootouts, and Devstral Small 2 at Q8 with six-figure context is a materially better coding agent than any starved 70B.
• The frontier squeeze: gpt-oss-120b (117B total, about 5B active, roughly 60GB of native MXFP4 weights) does not fit 48GB cleanly, but expert offload runs it at usable speeds with 128GB of system RAM behind it. Treat that as an experiment, not a daily driver; its clean homes are a single 80GB card or a 64GB-plus unified-memory Mac. Llama 4 Scout (109B, 17B active, roughly 60GB at Q4) sits in the same just-past-48GB purgatory.
• The serving move: 48GB serving a team runs a 24B to 32B AWQ model under vLLM with real batching:

vllm serve mistralai/Devstral-Small-2-24B-Instruct-2512 \
  --quantization awq --tensor-parallel-size 2 --max-model-len 131072

Above 48GB, be honest about the curve. The models that headline 2026, GLM-5.1 (roughly 750B MoE) and Kimi K2.6 (1T MoE, a reported 80.2 on SWE-bench Verified), want 8x H100 nodes at Q4. Between a dual-3090 tower and that world there is a thin band, 4x 3090 or RTX 6000-class cards, where KTransformers-style CPU-heavy MoE serving and ExLlamaV3 quad-GPU setups live, and it is genuinely fun territory, but the price-performance math of renting the big stuff usually wins past 96GB.

The picks at a glance

Quantization: what to actually download

The boring advice is correct: Q4_K_M is the default, and it is the right default. Spend spare VRAM on context before you spend it on Q5_K_M or Q6_K; the quality delta from a bigger quant is usually smaller than the utility delta from 4x the context. Q8_0 only makes sense when the model is so far under your budget that nothing else competes for the space, which is exactly the 27B-on-48GB case above.

Beyond the default, five notes earn their keep:

1. Prefer QAT and native low-bit builds when they exist. Gemma 4's quantization-aware-trained releases hold up better at 4-bit than post-hoc quants of the same weights, and gpt-oss ships in MXFP4 natively; requantizing it is pure loss.
2. IQ4_XS is the honest squeeze. When Q4_K_M misses your card by a gigabyte, IQ4_XS usually closes the gap with minimal damage. Below that, IQ3 and under, coding and long-form reasoning degrade visibly; describe that tier to yourself as "demo quality" and you will make better decisions.
3. Reasoning models are quant-sensitive. Chain-of-thought amplifies small errors across thousands of tokens. Keep R1-style and thinking-mode models one quant level higher than you would keep a chat model.
4. Quantize the KV cache carefully. q8_0 cache is nearly free; q4 cache on a long agent run is where mysterious quality loss comes from.
5. Get GGUFs from bartowski or Unsloth, whose day-one quants of new releases (including fixed chat templates, historically the most common source of "this model is broken" reports) have become the community standard. For vLLM, grab AWQ or FP8 builds instead; for TabbyAPI, EXL3.

Choosing without overthinking it

Work backwards from the job, not forwards from the leaderboard. If the machine is for chat, notes, extraction, and light code help, an 8GB card running Qwen3.5-9B is genuinely enough now, which was not true even a year ago. If you want one local daily driver, 16GB with gpt-oss-20b is the sweet spot of price to capability. If local agentic coding is the point, budget 24GB and run Qwen3.6-27B, with Devstral Small 2 as the specialist swap. Go multi-GPU only when you can say exactly which of the three things you are buying: bigger quants, longer context, or concurrent users. "A bigger model" stopped being the answer when the 27B class got this good.

Two things to watch through the rest of 2026: hybrid and linear attention spreading to every major family, which converts context from a VRAM problem into a solved one, and NVFP4-class 4-bit formats moving from datacenter inference into consumer stacks. Both trends push the same direction, and it is a pleasant one: the gap between what a rented H100 does and what the card already in your tower does keeps shrinking.