LMCache

Recomputing a long prompt's KV cache on every request is the tax LMCache removes. The library adds a cache management layer to LLM serving engines that persists key-value tensors across GPU memory, CPU RAM, local disk, and remote backends including Redis, Valkey, and S3-compatible object storage, then reloads them wherever the next request lands, cutting time-to-first-token sharply for long-context, multi-turn, RAG, and agentic workloads; recent releases report gains up to 10x on MoE-heavy inference. Integration is first class with vLLM and the vLLM production stack, SGLang, and NVIDIA's Dynamo datacenter framework, and hardware support spans NVIDIA, AMD MI300X, Arm, and Ascend. A pip install of lmcache plus a few lines of configuration attaches it to an existing deployment, with documentation maintained at docs.lmcache.ai. Governance moved under the PyTorch Foundation in October 2025, and the Apache 2.0 codebase has drawn over 11,000 GitHub stars, making it a default answer for teams whose GPU bills are dominated by redundant prefill.