{"id":"2028006725538967614","url":"https://x.com/agenticgirl/status/2028006725538967614","text":"🚨BREAKING: The open-source project that's saving enterprises millions in GPU costs.\n\nLMCache: 6.9K stars. 124K downloads/month. Used by Google Cloud, CoreWeave, and NVIDIA .\n\nHere's the problem nobody talks about:\n\nYour LLM inference is burning 50% of its compute on work it has already done.\n\nEvery time a user sends a message, your model recomputes the entire context from scratch. The system prompt. The conversation history. The retrieved documents.\n\nThat's thousands of GPU cycles. Wasted. Per request.\n\nLMCache makes the KV cache persistent and shareable across engine instances .\n\nInstead of the cache dying when a request finishes, LMCache offloads it to a shared layer:\n→ GPU (fastest) → CPU DRAM (fast, cheaper) → Local disk (slower, cheap) → S3/Network (slowest, cheapest) \n\nThis unlocks architecture patterns that were previously impossible:\n1./ Instant RAG \nProcess a 100-page PDF once. Store the KV cache. Now any user query against that doc starts instantly. Zero Time-To-First-Token.\n\n2./ Disaggregated Prefill \nRun heavy \"Prefill\" on H100s. Stream the cache to cheaper L4s for \"Decoding.\" Your expensive GPUs never sit idle.\n\n3./ Context Sharing \nMultiple users asking about the same context? Compute it once, serve everyone.\n\nThe results:\n🚀3-10x delay savings in multi-round QA workloads\n⚡️Dramatic reduction in Time-To-First-Token (TTFT) \n💰 GPU cycle reduction\n\nIntegrates directly with vLLM (72K stars) and SGLang (24K stars) .\n\nAdopted by: → GMI Cloud → Google Cloud → CoreWeave → Redis → NVIDIA Dynamo \n\nApache-2.0 license. pip install lmcache.\n\nStop letting your GPUs do the same homework twice.\n\nGitHub Repo in comments","author":{"name":"Sandhya","username":"agenticgirl","avatarUrl":"https://pbs.twimg.com/profile_images/2070871958519255040/_1ZRwhRv_200x200.jpg"},"createdAt":"Sun Mar 01 07:17:36 +0000 2026","engagement":{"replies":50,"retweets":317,"likes":2511,"views":142599},"media":{"photos":[{"url":"https://pbs.twimg.com/media/HCTtNyfboAAL9-G.jpg?name=orig","width":800,"height":1004}],"videos":[]}}