AWQ (Activation-aware Weight Quantization)
Reference implementation of activation-aware 4-bit and 3-bit weight quantization for LLMs and VLMs.
About
From MIT HAN Lab, AWQ is a post-training method that shrinks LLM and vision-language model weights to 4 or 3 bits without retraining. Its observation is that about one percent of weight channels matter most, and activation statistics from a small calibration set reveal which ones; scaling those channels up before rounding keeps accuracy far better than uniform rounding, with no backpropagation, which helps it generalize. The repository runs the scale search, writes real INT4 weights, and includes CUDA kernels for 4-bit weights with 16-bit activations plus TinyChat, a small runtime that demos chat and image or video reasoning on RTX desktops, laptops and Jetson Orin. Precomputed search results on Hugging Face cover Llama 2 and 3, Qwen 2.5, DeepSeek-R1 distills, VILA and NVILA. Setup means conda, an editable pip install, compiled kernels and FlashAttention, so a CUDA GPU is required. The work won the MLSys 2024 best paper award and is supported in Transformers, vLLM and TensorRT-LLM, where most users meet it. The code is MIT; the last commit landed in July 2025, so treat it as the research reference implementation.
Should you use AWQ (Activation-aware Weight Quantization)?
Pick it when
Use it when you need to ship a 4-bit LLM or vision-language model for GPU inference and want better quality than naive rounding at the same bit-width, especially if you plan to serve it with the paired TinyChat runtime.
Look elsewhere when
AWQ only compresses a finished model for inference; it does not fine-tune. For low-memory training use bitsandbytes with QLoRA, and for a maintained multi-method quantization pipeline that feeds vLLM, use llm-compressor.
Alternatives to AWQ (Activation-aware Weight Quantization)
- LLM Compressor
Applies AWQ, GPTQ, and fp8 or int8 schemes in one maintained library with vLLM-ready output, instead of research reference code.
- bitsandbytes
Quantizes at load time with no calibration pass and supports QLoRA training, but is not tuned for fast 4-bit serving like AWQ.
- llama.cpp
GGUF quantization runs on CPUs, Apple silicon, and many GPU backends, where AWQ checkpoints assume a GPU at inference time.
Reviews (0)
Leave a Review
No reviews yet. Be the first to review!
Details
- Category
- Model Training & Fine-Tuning
- Price
- Free
- Platform
- Local/Desktop
- Difficulty
- Intermediate (3/5)
- License
- MIT
- Minimum VRAM
- 8 GB
- Added
- Apr 3, 2026