14x faster and 90% cheaper LLM inference.

(100% open-source, KV cache management)

your LLM does the same expensive work over and over.

every request, it re-reads the same system prompts and the same documents from scratch, even if it processed them one second ago. token prices keep falling, but agent workloads re-send so much repeated context that the bill climbs anyway.

LMCache fixes this. it's an open-source KV cache management layer that plugs into vLLM, SGLang, and TensorRT-LLM.

here's how it works:

LLMs recompute their understanding of the same content on every request. the same system prompts, the same documents, processed from scratch every time, and a single GPU throws away roughly 15 TB of this reusable cache per day.

LMCache stores that cache and serves it back on repeat requests, running as a separate process completely outside the inference engine.

the engine just asks for the cache blocks it needs. LMCache handles all the heavy data movement across GPU, CPU, disk, and remote storage in parallel, so cache work never steals compute from inference.

it also reuses cache beyond exact prefixes. their CacheBlend technique (EuroSys 2025 best paper) keeps RAG documents cached no matter what order they appear in.

on H200s with a 235B model, that adds up to 14x faster time-to-first-token and 4x faster decoding. and since reuse skips the compute entirely (the same reason providers discount cached tokens by 90%), the cost savings follow directly.

GitHub repo: https://github.com/LMCache/LMCache

(don't forget to star 🌟)

i wrote a full breakdown of KV cache management that walks through why 𝗽𝗿𝗲𝗳𝗶𝘅 𝗰𝗮𝗰𝗵𝗶𝗻𝗴 silently breaks in three common cases, the 𝗱𝗶𝘀𝗮𝗴𝗴𝗿𝗲𝗴𝗮𝘁𝗲𝗱 𝗮𝗿𝗰𝗵𝗶𝘁𝗲𝗰𝘁𝘂𝗿𝗲 behind the 14x speedup, 𝗖𝗮𝗰𝗵𝗲𝗕𝗹𝗲𝗻𝗱, and how to turn every document in your knowledge base into a reusable cached asset.

the article is quoted below.

ガチでこれは強い。
海外のエンジニアが、Claude Code内でFable 5(Claudeの最上位モデル)からGPT 5.5搭載のCodexを必要な時だけ呼び出す構成を試した。結果、24時間で20億トークンを使っても、Fable 5の週間上限は49%・Codexは30%しか減らず、体感3倍の仕事が終わったと報告してる。

・Fable 5が指示出し・レビューだけ担当
・重い実装はCodex側の別枠で処理
・2つのサブスク上限を同時に使い切れる

信じてほしいとまで言ってる👇

📸 A quiet lace dream from above!

Made a set of “Lace Dream Portraits” with ChatGPT-Image2.

White lace bedding, soft satin dresses, scattered hair, and top-down composition.

The focus is on natural fabric support, body lines, soft light, and a more cinematic portrait mood.

Prompt below 👇