Univer is an open-source SDK for embedding spreadsheet, document, and presentation editing directly into your own product or server-side workflow.
- Canvas-based rendering and a dedicated formula engine for complex workbooks - Plugin architecture for composing, replacing, or lazy-loading capabilities - One Facade API that works in the browser and on Node.js - Extend through custom plugins, commands, services, and UI components
Amber is a high-level language that compiles to Bash, making shell scripts easier to write and maintain.
- Compiles directly to Bash for compatibility with existing environments - Designed specifically for cloud service scripting - Offers high-level abstractions over shell complexity - Supported by documentation and community channels (Discord, IRC, Matrix)
"Our model generates 100 tokens in 42 seconds. How do you make it 5x faster?"
You: "I'll optimize the model architecture and use a better GPU."
Interview over.
Here's what you missed:
The real bottleneck isn't compute. It's redundant computation.
Without KV caching, your model recalculates the same attention keys and values for every single token generation.
That's why a 9-second inference becomes 42 seconds. You're wasting 80% of your time on repeated calculations.
The fundamental issue:
(refer image below as you read ahead)
LLM token generation is autoregressive:
- Generate token 1 from the prompt - Generate token 2 from prompt + token 1 - Generate token 3 from prompt + token 1 + token 2
At each step, you're reprocessing ALL previous tokens through attention.
Token 50? You've computed attention for token 1 fifty times.
The reality of attention mechanism:
For each token, the transformer computes:
- Query (Q) from current token - Key (K) from all previous tokens - Value (V) from all previous tokens
Then: Attention(Q, K, V) = softmax(QK^T)V
Problem: K and V for previous tokens never change. You're recalculating identical matrices every single step.
How KV caching solves this:
Instead of recomputing K and V matrices:
- Cache them after first computation - Reuse cached values for subsequent tokens - Only compute K and V for the new token
Without KV caching (token 50): - Compute Q, K, V for all 50 tokens → O(n²)
With KV caching (token 50): - Load cached K, V for tokens 1-49 - Compute Q, K, V only for token 50 → O(n)
You've eliminated quadratic redundancy.
So what's the tradeoff:
While KV caching makes the inference faster, it also takes up a lot of memory, so there is always a tradeoff between speed and memory.
Why your first token always takes longer:
KV caching speeds up inference by computing the prompt's KV cache before generating tokens.
This is exactly why ChatGPT takes longer to generate the first token than the rest.
First token: Computing KV cache for entire prompt
Remaining tokens: Just loading cached KVs + computing new token
---- Everything above is how KV caching works inside a single request.
Running it in production is a different problem. Caches break on document reordering, and a single GPU throws away roughly 15 TB of reusable cache per day.
I wrote an article that picks up exactly where this post ends: how a new open-source architecture manages KV cache at production scale, with 14x faster time-to-first-token to show for it.