LMDeploy

LMDeploy comes from the InternLM team as a toolkit for compressing, deploying, and serving large language models. It ships two inference engines: TurboMind, a high-performance CUDA engine with persistent batching, blocked KV cache, and optimized kernels, and a PyTorch engine aimed at flexibility and easier experimentation. Quantization support covers weight-only quantization and KV cache quantization, and the project reports up to 1.8 times higher request throughput than vLLM, with 4-bit inference running about 2.4 times faster than FP16. Coverage is broad: more than 40 LLM architectures including Llama, Qwen, DeepSeek, Gemma, and Phi, plus over 30 vision-language models such as InternVL, LLaVA, and Qwen-VL, spanning roughly 1B to 754B parameters. Multi-machine multi-GPU deployment, tensor parallelism, and automatic prefix caching round out the serving features. The toolkit is open source under Apache 2.0 and is used by teams putting LLM and VLM inference into production.