Prefill/decode | Blog

Inference2026-07-1613 min

How NVIDIA Dynamo Runs an Inference Fleet

Past a certain load one GPU node isn't enough, and a bigger GPU won't save you — the fix is a fleet. NVIDIA Dynamo is an open framework for running one, and its design is the clearest overview I've found of the whole problem: three planes — request, control, storage & events — and three control loops that keep the fleet fast, right-sized, and alive. These are my notes on the parts a Dynamo fleet is built from, how it manages them, and the one metric — goodput — that judges the result.

Distributed inference Serving Prefill/decode Dynamo

LLM Inference: The Life of a Request

A serving model does two very different kinds of work for every request: it reads the prompt, then writes a reply. Those two phases — prefill and decode — have opposite bottlenecks, and almost every confusing thing about inference performance comes from treating them as one. This is the mental model the rest of my inference writing builds on.