Tag

Prefill/decode

Inference2026-07-1610 min

Distributed Inference: From One GPU to a Fleet

Past a certain load you run out of node, and the fix isn't a bigger GPU — it's a fleet, with a cache-aware router in front and a shared KV store behind. The layout: the boxes a production inference fleet is made of, the data-plane/control-plane split, the three walls that force the jump, and the one metric — goodput — that judges the whole thing.

Distributed inference Serving Prefill/decode

Inference2026-06-249 min

LLM Inference: The Life of a Request

A serving model does two very different kinds of work for every request: it reads the prompt, then writes a reply. Those two phases — prefill and decode — have opposite bottlenecks, and almost every confusing thing about inference performance comes from treating them as one. This is the mental model the rest of my inference writing builds on.

Transformer Prefill/decode KV cache