LLM 推理提速 14 倍，成本降低 90%。
(100% 开源，KV cache 管理)
你的 LLM 一直在重复做同样的高昂工作。
每一次请求，它都会从头重新读取相同的系统提示词和文档，哪怕一秒钟前刚处理过。虽然 token 价格在下降，但智能体工作负载发送了太多重复上下文，导致账单依然在飙升。
LMCache 解决了这个问题。它是一个开源的 KV cache 管理层，可以接入 vLLM、SGLang 和 TensorRT-LLM。
原理如下：
LLM 在每次请求时都会重新计算对相同内容的理解。同样的系统提示词、同样的文档，每次都从头处理，单张 GPU 每天大约会丢弃 15 TB 这种可重复使用的缓存。
LMCache 存储这些缓存并在重复请求时提供，它作为一个完全独立于推理引擎的进程运行。
引擎只需请求所需的缓存块。LMCache 并行处理 GPU、CPU、磁盘和远程存储之间的所有重型数据传输，因此缓存工作永远不会占用推理的计算资源。
它还能在精确前缀之外复用缓存。他们的 CacheBlend 技术（EuroSys 2025 最佳论文）无论 RAG 文档以什么顺序出现，都能保持缓存。
在 H200 上运行 235B 模型时，首字延迟 (TTFT) 缩短了 14 倍，解码速度提升了 4 倍。由于复用完全跳过了计算（这也是供应商对缓存 token 给予 90% 折扣的原因），成本节省是立竿见影的。
GitHub 仓库：https://github.com/LMCache/LMCache
(别忘了点个 star 🌟)
我写了一篇关于 KV cache 管理的深度解析，详细介绍了为什么前缀缓存 (𝗽𝗿𝗲𝗳𝗶𝘅 𝗰𝗮𝗰𝗵𝗶𝗻𝗴) 在三种常见情况下会失效、实现 14 倍提速背后的解耦架构 (𝗱𝗶𝘀𝗮𝗴𝗴𝗿𝗲𝗴𝗮𝘁𝗲𝗱 𝗮𝗿𝗰𝗵𝗶𝘁𝗲𝗰𝘁𝘂𝗿𝗲)、𝗖𝗮𝗰𝗵𝗲𝗕𝗹𝗲𝗻𝗱，以及如何将知识库中的每个文档都变成可复用的缓存资产。
文章引用如下。
(100% 开源，KV cache 管理)
你的 LLM 一直在重复做同样的高昂工作。
每一次请求，它都会从头重新读取相同的系统提示词和文档，哪怕一秒钟前刚处理过。虽然 token 价格在下降，但智能体工作负载发送了太多重复上下文，导致账单依然在飙升。
LMCache 解决了这个问题。它是一个开源的 KV cache 管理层，可以接入 vLLM、SGLang 和 TensorRT-LLM。
原理如下：
LLM 在每次请求时都会重新计算对相同内容的理解。同样的系统提示词、同样的文档，每次都从头处理，单张 GPU 每天大约会丢弃 15 TB 这种可重复使用的缓存。
LMCache 存储这些缓存并在重复请求时提供，它作为一个完全独立于推理引擎的进程运行。
引擎只需请求所需的缓存块。LMCache 并行处理 GPU、CPU、磁盘和远程存储之间的所有重型数据传输，因此缓存工作永远不会占用推理的计算资源。
它还能在精确前缀之外复用缓存。他们的 CacheBlend 技术（EuroSys 2025 最佳论文）无论 RAG 文档以什么顺序出现，都能保持缓存。
在 H200 上运行 235B 模型时，首字延迟 (TTFT) 缩短了 14 倍，解码速度提升了 4 倍。由于复用完全跳过了计算（这也是供应商对缓存 token 给予 90% 折扣的原因），成本节省是立竿见影的。
GitHub 仓库：https://github.com/LMCache/LMCache
(别忘了点个 star 🌟)
我写了一篇关于 KV cache 管理的深度解析，详细介绍了为什么前缀缓存 (𝗽𝗿𝗲𝗳𝗶𝘅 𝗰𝗮𝗰𝗵𝗶𝗻𝗴) 在三种常见情况下会失效、实现 14 倍提速背后的解耦架构 (𝗱𝗶𝘀𝗮𝗴𝗴𝗿𝗲𝗴𝗮𝘁𝗲𝗱 𝗮𝗿𝗰𝗵𝗶𝘁𝗲𝗰𝘁𝘂𝗿𝗲)、𝗖𝗮𝗰𝗵𝗲𝗕𝗹𝗲𝗻𝗱，以及如何将知识库中的每个文档都变成可复用的缓存资产。
文章引用如下。
14x faster and 90% cheaper LLM inference.
(100% open-source, KV cache management)
your LLM does the same expensive work over and over.
every request, it re-reads the same system prompts and the same documents from scratch, even if it processed them one second ago. token prices keep falling, but agent workloads re-send so much repeated context that the bill climbs anyway.
LMCache fixes this. it's an open-source KV cache management layer that plugs into vLLM, SGLang, and TensorRT-LLM.
here's how it works:
LLMs recompute their understanding of the same content on every request. the same system prompts, the same documents, processed from scratch every time, and a single GPU throws away roughly 15 TB of this reusable cache per day.
LMCache stores that cache and serves it back on repeat requests, running as a separate process completely outside the inference engine.
the engine just asks for the cache blocks it needs. LMCache handles all the heavy data movement across GPU, CPU, disk, and remote storage in parallel, so cache work never steals compute from inference.
it also reuses cache beyond exact prefixes. their CacheBlend technique (EuroSys 2025 best paper) keeps RAG documents cached no matter what order they appear in.
on H200s with a 235B model, that adds up to 14x faster time-to-first-token and 4x faster decoding. and since reuse skips the compute entirely (the same reason providers discount cached tokens by 90%), the cost savings follow directly.
GitHub repo: https://github.com/LMCache/LMCache
(don't forget to star 🌟)
i wrote a full breakdown of KV cache management that walks through why 𝗽𝗿𝗲𝗳𝗶𝘅 𝗰𝗮𝗰𝗵𝗶𝗻𝗴 silently breaks in three common cases, the 𝗱𝗶𝘀𝗮𝗴𝗴𝗿𝗲𝗴𝗮𝘁𝗲𝗱 𝗮𝗿𝗰𝗵𝗶𝘁𝗲𝗰𝘁𝘂𝗿𝗲 behind the 14x speedup, 𝗖𝗮𝗰𝗵𝗲𝗕𝗹𝗲𝗻𝗱, and how to turn every document in your knowledge base into a reusable cached asset.
the article is quoted below.
(100% open-source, KV cache management)
your LLM does the same expensive work over and over.
every request, it re-reads the same system prompts and the same documents from scratch, even if it processed them one second ago. token prices keep falling, but agent workloads re-send so much repeated context that the bill climbs anyway.
LMCache fixes this. it's an open-source KV cache management layer that plugs into vLLM, SGLang, and TensorRT-LLM.
here's how it works:
LLMs recompute their understanding of the same content on every request. the same system prompts, the same documents, processed from scratch every time, and a single GPU throws away roughly 15 TB of this reusable cache per day.
LMCache stores that cache and serves it back on repeat requests, running as a separate process completely outside the inference engine.
the engine just asks for the cache blocks it needs. LMCache handles all the heavy data movement across GPU, CPU, disk, and remote storage in parallel, so cache work never steals compute from inference.
it also reuses cache beyond exact prefixes. their CacheBlend technique (EuroSys 2025 best paper) keeps RAG documents cached no matter what order they appear in.
on H200s with a 235B model, that adds up to 14x faster time-to-first-token and 4x faster decoding. and since reuse skips the compute entirely (the same reason providers discount cached tokens by 90%), the cost savings follow directly.
GitHub repo: https://github.com/LMCache/LMCache
(don't forget to star 🌟)
i wrote a full breakdown of KV cache management that walks through why 𝗽𝗿𝗲𝗳𝗶𝘅 𝗰𝗮𝗰𝗵𝗶𝗻𝗴 silently breaks in three common cases, the 𝗱𝗶𝘀𝗮𝗴𝗴𝗿𝗲𝗴𝗮𝘁𝗲𝗱 𝗮𝗿𝗰𝗵𝗶𝘁𝗲𝗰𝘁𝘂𝗿𝗲 behind the 14x speedup, 𝗖𝗮𝗰𝗵𝗲𝗕𝗹𝗲𝗻𝗱, and how to turn every document in your knowledge base into a reusable cached asset.
the article is quoted below.
16
47
325
27.4K
