Tag
4 posts· All tags
Orca reported 36.9× the throughput of FasterTransformer at the same latency without touching the model's math; a year later vLLM's profiling found as little as a fifth of KV-cache memory holding actual tokens. Two papers, two mechanisms — re-form the batch every iteration, page the cache in 16-token blocks — and everything else an LLM scheduler does today is policy.
The KV cache is num_layers × num_heads × 2 × seq_len × d_head × bytes, and every way to shrink attention attacks one of those terms. MQA and GQA share keys across heads, MLA compresses them to a latent that beats full attention on quality, FlashAttention computes the exact same thing with a fraction of the memory traffic, and the frontier goes after the rest — layers, sparsity, and the quadratic itself.
Attention is the one operation that lets a token look back at the whole sequence — and it is also where the KV cache comes from. This post derives scaled dot-product attention from the ground up, then watches the cache, and the memory wall from the last post, fall straight out of the math. The first of two; the next is about making it cheap.
A serving model does two very different kinds of work for every request: it reads the prompt, then writes a reply. Those two phases — prefill and decode — have opposite bottlenecks, and almost every confusing thing about inference performance comes from treating them as one. This is the mental model the rest of my inference writing builds on.