You're in an ML Engineer interview at Anthropic.

The interviewer asks:

"Our model generates 100 tokens in 42 seconds. How do you make it 5x faster?"

You: "I'll optimize the model architecture and use a better GPU."

Interview over.

Here's what you missed:

The real bottleneck isn't compute. It's redundant computation.

Without KV caching, your model recalculates the same attention keys and values for every single token generation.

That's why a 9-second inference becomes 42 seconds. You're wasting 80% of your time on repeated calculations.

The fundamental issue:

(refer image below as you read ahead)

LLM token generation is autoregressive:

- Generate token 1 from the prompt
- Generate token 2 from prompt + token 1
- Generate token 3 from prompt + token 1 + token 2

At each step, you're reprocessing ALL previous tokens through attention.

Token 50? You've computed attention for token 1 fifty times.

The reality of attention mechanism:

For each token, the transformer computes:

- Query (Q) from current token
- Key (K) from all previous tokens
- Value (V) from all previous tokens

Then: Attention(Q, K, V) = softmax(QK^T)V

Problem: K and V for previous tokens never change. You're recalculating identical matrices every single step.

How KV caching solves this:

Instead of recomputing K and V matrices:

- Cache them after first computation
- Reuse cached values for subsequent tokens
- Only compute K and V for the new token

Without KV caching (token 50):
- Compute Q, K, V for all 50 tokens → O(n²)

With KV caching (token 50):
- Load cached K, V for tokens 1-49
- Compute Q, K, V only for token 50 → O(n)

You've eliminated quadratic redundancy.

So what's the tradeoff:

While KV caching makes the inference faster, it also takes up a lot of memory, so there is always a tradeoff between speed and memory.

Why your first token always takes longer:

KV caching speeds up inference by computing the prompt's KV cache before generating tokens.

This is exactly why ChatGPT takes longer to generate the first token than the rest.

First token: Computing KV cache for entire prompt

Remaining tokens: Just loading cached KVs + computing new token

----
Everything above is how KV caching works inside a single request.

Running it in production is a different problem. Caches break on document reordering, and a single GPU throws away roughly 15 TB of reusable cache per day.

I wrote an article that picks up exactly where this post ends: how a new open-source architecture manages KV cache at production scale, with 14x faster time-to-first-token to show for it.

The article is quoted below.