Running LLMs Without Nvidia in 2026: ROCm and Apple Silicon

AMD's ROCm stack and Apple Silicon are the two real exits from Nvidia for local LLM work in 2026, and they solve different problems. This is a field guide to both as of August 2026: which runtimes actually work, what throughput real hardware delivers, and the porting traps that still separate 'it runs' from 'it runs well'. It is written for developers deciding where inference money should go, not for spec-sheet tourists.

Two exits, two philosophies

The AMD exit is CUDA-shaped. ROCm imitates the CUDA stack closely enough that PyTorch on an AMD GPU still exposes the device through torch.cuda, HIP is close to a mechanical port of CUDA C++, and the serving layer is the same software you would run on H100s. AMD competes on serving throughput and memory per dollar, and its story splits sharply between data center CDNA silicon (Instinct MI300X through MI355X) and consumer RDNA silicon (Radeon RX 7000 and 9000).

The Apple exit is not CUDA-shaped at all. There is no discrete GPU and no VRAM boundary, and the native framework, MLX, is its own NumPy-like array framework built around lazy evaluation and unified memory. Apple competes on memory capacity per watt and single-user latency. Nothing else puts half a terabyte of model weights on one desk at under 200W.

Three shifts over the past year moved both paths from hobbyist to defensible:

• AMD shipped ROCm 7.14 on July 15, 2026, its first production release built end to end by TheRock, its new open source build and release system, with modular SDK installs and official coverage spanning Instinct MI350-series, Radeon RX 7000 and 9000, and Ryzen AI Max APUs on both Linux and Windows.
• Official vLLM ROCm images now ship on the upstream Docker Hub as vllm/vllm-openai-rocm, in stable and nightly tags, and the older AMD-maintained rocm/vllm images are formally deprecated. Not long ago, AMD support meant hunting for the right fork.
• Ollama announced on March 30, 2026 that it is moving its Apple Silicon backend to MLX. Current builds still route GGUF pulls through llama.cpp and send safetensors models to the new MLX runner, which is labeled a preview and supports a handful of architectures so far.

If you tried any of this in 2024 and walked away, the terrain has genuinely changed. It has not flattened, though, and most of this article is about where the cliffs still are.

Memory bandwidth sets your ceiling

Token generation is memory-bandwidth-bound: every generated token reads all active weights. A dense 8B model at 4-bit is roughly 4.5GB of weights, so a 960 GB/s card has a theoretical ceiling around 200 t/s on it regardless of compute, and real kernels land well below theory. Prefill (prompt ingestion) is the opposite, compute-bound, which matters later when we get to Apple's historic weak spot.

The pattern to internalize: discrete Radeon gives you bandwidth but limited capacity, while Strix Halo and Apple give you capacity at lower bandwidth. That is exactly the trade mixture-of-experts models exploit. A MoE like Qwen3-30B-A3B holds 30B parameters but activates about 3B per token, so it wants lots of memory and only moderate bandwidth. This one fact drives most sensible off-Nvidia purchases in 2026.

ROCm in 2026: finally boring, mostly

The ROCm 7 era fixed the two things that made AMD a punchline: install pain and framework lag. One installer covers Linux and Windows, RDNA 4 (RX 9000 series) is officially supported, and the 7.14 release in July 2026 moved the whole stack onto TheRock's automated builds with modular core and domain SDKs. On Windows, llama.cpp and Ollama both run well on Radeon; vLLM remains Linux-only. AMD now publishes official ROCm-enabled PyTorch wheels for Windows covering RX 7000 and 9000 plus select Ryzen AI APUs, but Linux is still where the full stack and the tuned kernels live, so treat it as the serious path.

Fine-tuning stopped being Nvidia-exclusive too. Unsloth added official AMD support spanning consumer Radeon, Instinct accelerators, and Ryzen AI Max, on Windows, WSL, and Linux. The AMD path rests on bitsandbytes ROCm support for QLoRA, HIP and Triton kernels in place of CUDA-only ones, and fallback to xformers where FlashAttention 2 is unavailable.

The honest caveats: community reports still put PyTorch-on-ROCm performance a measurable step behind comparable CUDA setups, with the size of the gap swinging by workload, and the CDNA versus RDNA split is real. Instinct cards get the tuned kernels first; consumer Radeon gets them later or via Triton substitutes.

llama.cpp on Radeon: try Vulkan before you fight HIP

The least intuitive advice in this article: on consumer RDNA 3 cards, llama.cpp's Vulkan backend frequently beats the dedicated ROCm HIP backend. Community benchmarks show a 7900 XTX generating around 190 t/s on Llama 2 7B Q4_0 through Vulkan, while recurring community reports have the same card landing meaningfully lower through HIP on the same workload. The HIP gap is tracked as an open llama.cpp issue, not your build being broken. Vulkan also needs no ROCm install at all, which is why KoboldCpp and LM Studio lean on Vulkan for AMD by default, with ROCm as a selectable runtime.

Both builds are one cmake invocation:

# Vulkan: no ROCm required, works on Windows and Linux
cmake -B build -DGGML_VULKAN=ON
cmake --build build --config Release -j

# HIP/ROCm: Linux, RDNA 3 example (gfx1100 = RX 7900 series)
HIPCXX="$(hipconfig -l)/clang" HIP_PATH="$(hipconfig -R)" cmake -S . -B build -DGGML_HIP=ON -DGPU_TARGETS=gfx1100 -DCMAKE_BUILD_TYPE=Release
cmake --build build -j 16

Then serve as usual with llama-server -m model.gguf -ngl 99. Benchmark both backends on your own card with llama-bench before committing; the ranking flips depending on model, quant, and driver version, and prompt processing in particular can favor HIP even where generation favors Vulkan.

Serving on AMD: vLLM is real, with an asterisk

For production-style serving, vLLM on ROCm is now a supported first-class target rather than a science project. The upstream vllm/vllm-openai-rocm images cover Instinct MI200 (gfx90a), MI300 (gfx942), and MI350 (gfx950), plus Radeon RX 7900 series (gfx1100/1101), RX 9000 series (gfx1200/1201), and Ryzen AI Max APUs (gfx1150/1151):

docker pull vllm/vllm-openai-rocm:latest
docker run -it --device=/dev/kfd --device=/dev/dri --group-add video vllm/vllm-openai-rocm:latest vllm serve meta-llama/Llama-3.1-8B-Instruct

The asterisk is the consumer tier. On gfx1100 you still hit caveats like building with FlashAttention disabled because the tuned attention kernels target Instinct, and AMD's performance-tuned configurations are published for Instinct and, more recently, Radeon on Linux. SGLang is in a similar place: officially in AMD's supported framework list, clearly aimed at MI300-class deployments. The practical read: serve production traffic from Instinct, use Radeon vLLM for development parity, and keep llama.cpp as the consumer-card workhorse. There is still no AMD equivalent of the deeply tuned closed-loop stack Nvidia users get from TensorRT-LLM, but for most teams vLLM's ROCm images are close enough.

Strix Halo: the 128GB wildcard

The Ryzen AI Max+ 395 (Strix Halo), the chip inside Framework Desktop and a wave of mini PCs, is the most interesting AMD purchase for local LLMs and the most misunderstood. It pairs 16 Zen 5 cores and a 40 CU RDNA 3.5 iGPU with up to 128GB of unified LPDDR5X at roughly 256 GB/s. That bandwidth number tells you everything: community benchmarks put dense 70B models in the single-digit t/s range, which is unusable for interactive work.

MoE models are a different story. Community numbers put Qwen3-30B-A3B around 50 t/s on Linux with the Vulkan or ROCm backend, and 120B-class MoE models like gpt-oss-120b roughly in the 30 to 50 t/s band, because only a few billion parameters activate per token while the full weights sit comfortably in 128GB. If your 2026 model diet is MoE, a Strix Halo box is the cheapest way to hold 100B-plus weights locally. If it is dense models, buy bandwidth instead.

Apple Silicon: MLX won

The framework question on Mac is settled. Apple threw its weight behind MLX at WWDC 2025, Ollama is migrating its Apple backend to it, and LM Studio ships it alongside llama.cpp as one of its two bundled engines. In community benchmarks, MLX typically leads llama.cpp's Metal backend in decode throughput on identical hardware, often by a wide margin. One widely shared benchmark had Qwen3-Coder-30B-A3B generating roughly three times faster via MLX on an M4 Pro than through Ollama's llama.cpp path on the same machine, with raw llama.cpp landing in between, which says as much about wrapper overhead as about engines: measure your stack end to end, not just the kernel.

Getting started is two commands, and the mlx-community organization on Hugging Face maintains thousands of pre-quantized conversions:

pip install mlx-lm
mlx_lm.generate --prompt "How tall is Mt Everest?"
# OpenAI-compatible local server
mlx_lm.server --model mlx-community/Llama-3.2-3B-Instruct-4bit --port 8080

MLX's historic weakness is prefill. Generation is bandwidth-bound and Macs have bandwidth; prompt processing is compute-bound and Mac GPUs gave up a lot of FLOPS to Nvidia. On long prompts, older M-series chips can spend the vast majority of wall time on prefill. This is exactly what the M5's GPU neural accelerators attack: Apple's own published MLX benchmarks on long prompts measured time-to-first-token several times faster than M4 across small and mid-size models, with generation modestly faster on the back of a memory bandwidth bump. If long-context work on a Mac frustrated you before, the M5 generation is the fix, and MLX is the framework those accelerators were built for.

Big models and Mac clusters

The M3 Ultra Mac Studio with 512GB of unified memory remains the singular party trick of the Apple path: in widely shared community runs it holds DeepSeek R1 671B in 4-bit and generates at 17 to 18 t/s while drawing under 200W, something no single consumer Nvidia configuration can do at any price near its roughly 10,000 dollar cost. You have to manually raise the GPU wired memory limit first, since macOS reserves a slice of unified memory for the system:

# allow the GPU to wire ~448GB on a 512GB machine
sudo sysctl iogpu.wired_limit_mb=458752

Past one box, exo pools multiple Macs into a single inference cluster with pipeline and tensor parallelism, is open source under Apache 2.0, and remains in active development through 2026. Its headline feature is RDMA over Thunderbolt 5 on macOS 26.2 and later, sharply cutting inter-node latency, and the project's own benchmarks show DeepSeek v3.1 671B running across four 512GB M3 Ultra Mac Studios with tensor parallelism. Temper expectations: these clusters excel at fitting huge models, not at batch throughput, and TTFT on huge dense contexts is still a weak point.

Which runtime where

The porting traps that still bite

These are the failure modes that eat weekends, collected from the issues above rather than from vendor slides.

1. The flash-attn hard pin. Half the LLM repos on GitHub pin flash-attn in requirements, and the upstream package is CUDA-only. The AMD answer is the ROCm FlashAttention fork, with Composable Kernel builds for Instinct and a Triton-based backend for consumer RDNA, or letting a framework like Unsloth fall back to xformers automatically. Budget time for this dependency specifically.
2. Stale bitsandbytes folklore. Guides from 2024 say bitsandbytes does not work on AMD. Mainline bitsandbytes now ships prebuilt ROCm wheels on PyPI, with all features supported on both consumer RDNA (gfx1100 included) and data center CDNA, on Linux and Windows alike. Check your installed version before believing an old blog post.
3. Quant ecosystems are CUDA-first. AWQ and GPTQ kernels arrived on ROCm later and are tuned for Nvidia tensor shapes. On Radeon, GGUF quants through llama.cpp are the path of least resistance; on Mac, MLX has its own quant format, so teams supporting both maintain two artifact sets per model.
4. It compiles, but it is slow. HIP will happily build kernels tuned for Nvidia's warp size of 32 and tensor core shapes; RDNA runs wave32/wave64 with different matrix acceleration, and Triton autotuning configs shipped for A100/H100 rarely fit. The visible symptom is exactly what llama.cpp users see: a portable Vulkan backend outrunning the vendor stack on RDNA 3.
5. The torch.cuda illusion. On ROCm, PyTorch reports the AMD GPU as a CUDA device, so most pure PyTorch code just works, right up until a dependency ships a prebuilt CUDA wheel, inline PTX, or a cpp_extension build. Grep a project for .cu files and custom ops before promising a port date.
6. Silent CPU fallback on Mac. For PyTorch on Apple Silicon, PYTORCH_ENABLE_MPS_FALLBACK=1 keeps unsupported ops from crashing by quietly running them on CPU. Correct results, catastrophic speed. For inference on Mac, use MLX or llama.cpp; treat torch MPS as a compatibility layer, not a runtime.
7. No vLLM-class serving on Mac GPUs. vLLM has no Metal backend, so the Mac serving story is mlx_lm.server or llama-server. Fine for a team dev box, not a substitute for continuous-batching throughput. If you need real concurrency off Nvidia, that is what Instinct plus vLLM is for.

How to choose

• Interactive daily driver: an M4 Max class Mac with 64 to 128GB and MLX. Best latency per watt, best software momentum on the Apple side, and the M5 generation removes the long-context penalty.
• Budget 24GB inference box: a used RX 7900 XTX running llama.cpp on Vulkan. Skip the ROCm install entirely until a workload demands PyTorch.
• Biggest models per dollar: Strix Halo with 128GB for MoE-heavy diets, or a 512GB M3 Ultra if a 671B-class model on one box justifies five figures.
• Production serving without Nvidia: Instinct MI300X/MI355X with the official vLLM or SGLang ROCm images. Consumer Radeon and Macs are not that tier.
• Fine-tuning: AMD is finally viable via Unsloth with QLoRA on both Radeon and Instinct; on Mac, mlx-lm's LoRA path is the pragmatic route.

What to watch next: Ollama's MLX backend leaving preview, the open llama.cpp HIP performance gap on RDNA closing, M5 Max and Ultra silicon bringing neural accelerators to the big-memory tiers, and whether TheRock cadence keeps consumer cards inside the support matrix from day one. The CUDA moat is now a kernel-tuning moat rather than a functionality moat, and those drain faster.