Tag
4 posts· All tags
The router in DeepSeek-V3 is a 7,168 × 256 matrix — 1.8M parameters picking 8 of 256 experts for every token, in a layer of 11.3B. This post covers what the MLP post's one MoE paragraph skipped: how the router works, what the sparsity buys, and the recipe the mid-2026 frontier — DeepSeek-V4, GLM-5.2, Kimi K2.6 — converged on.
In a Llama 2 70B layer the attention projections hold 151M parameters and the MLP next to them holds 704.6M — 82% of the layer, roughly 80% of the model, and most of the bytes decode streams for every token. This post builds the block the attention posts walked past: two matmuls and a gate, why the transformer collapses without it, and where the frontier is taking it.
Attention is the one operation that lets a token look back at the whole sequence — and it is also where the KV cache comes from. This post derives scaled dot-product attention from the ground up, then watches the cache, and the memory wall from the last post, fall straight out of the math. The first of two; the next is about making it cheap.
A serving model does two very different kinds of work for every request: it reads the prompt, then writes a reply. Those two phases — prefill and decode — have opposite bottlenecks, and almost every confusing thing about inference performance comes from treating them as one. This is the mental model the rest of my inference writing builds on.