PowerInfer

Started at the IPADS lab of Shanghai Jiao Tong University, PowerInfer speeds up local LLM inference by exploiting how, in models with ReLU-style activations, a small set of hot neurons fire for almost every token while most fire rarely. Offline profiling finds the hot set, which stays in GPU memory, the CPU handles the rest, and small learned predictors skip neurons expected to stay idle. The authors measured up to about 11x the speed of llama.cpp for a ReLU Falcon 40B on one RTX 4090. The engine is a C++ fork of llama.cpp with CUDA and ROCm backends, built with CMake on Linux or Windows machines with AVX2, and it reads special PowerInfer GGUF files bundling predictor weights; published ones cover ReLU Llama 2, Falcon 40B, ProSparse Llama 2 and Bamboo 7B, while ordinary GGUF models run without the gain. A --vram-budget flag caps GPU use, and a smallthinker folder adds an engine for SmallThinker sparse MoE models on phones and boards like Raspberry Pi 5. MIT licensed. The last code change landed in July 2025 and the repo moved under Tiiny AI's organization, so treat it as an experimenter's tool, not a production server.