🚨BREAKING: The open-source project that's saving enterprises millions in GPU costs.
LMCache: 6.9K stars. 124K downloads/month. Used by Google Cloud, CoreWeave, and NVIDIA .
Here's the problem nobody talks about:
Your LLM inference is burning 50% of its compute on work it has already done.
Every time a user sends a message, your model recomputes the entire context from scratch. The system prompt. The conversation history. The retrieved documents.
That's thousands of GPU cycles. Wasted. Per request.
LMCache makes the KV cache persistent and shareable across engine instances .
Instead of the cache dying when a request finishes, LMCache offloads it to a shared layer:
→ GPU (fastest) → CPU DRAM (fast, cheaper) → Local disk (slower, cheap) → S3/Network (slowest, cheapest)
This unlocks architecture patterns that were previously impossible:
1./ Instant RAG
Process a 100-page PDF once. Store the KV cache. Now any user query against that doc starts instantly. Zero Time-To-First-Token.
2./ Disaggregated Prefill
Run heavy "Prefill" on H100s. Stream the cache to cheaper L4s for "Decoding." Your expensive GPUs never sit idle.
3./ Context Sharing
Multiple users asking about the same context? Compute it once, serve everyone.
The results:
🚀3-10x delay savings in multi-round QA workloads
⚡️Dramatic reduction in Time-To-First-Token (TTFT)
💰 GPU cycle reduction
Integrates directly with vLLM (72K stars) and SGLang (24K stars) .
Adopted by: → GMI Cloud → Google Cloud → CoreWeave → Redis → NVIDIA Dynamo
Apache-2.0 license. pip install lmcache.
Stop letting your GPUs do the same homework twice.
GitHub Repo in comments
