Tag

Distributed inference

Inference2026-08-3112 min

Dynamo's KV Router: Routing on What the Fleet Already Cached

A fleet's replicas are not interchangeable — each one holds a different slice of KV cache, and routing to the wrong one turns a cache hit into a full prefill. NVIDIA Dynamo's KV router answers that with one number per worker: a cost combining how much of the prompt that worker already has against how loaded it is. These are my notes on the parts that produce those two numbers — block hashes and KV events, the global prefix index, the tiered block manager behind it — and on the cost function that spends them.

Distributed inference Serving Routing KV cache Dynamo

Inference2026-07-1613 min

How NVIDIA Dynamo Runs an Inference Fleet

Past a certain load one GPU node isn't enough, and a bigger GPU won't save you — the fix is a fleet. NVIDIA Dynamo is an open framework for running one, and its design is the clearest overview I've found of the whole problem: three planes — request, control, storage & events — and three control loops that keep the fleet fast, right-sized, and alive. These are my notes on the parts a Dynamo fleet is built from, how it manages them, and the one metric — goodput — that judges the result.

Distributed inference Serving Prefill/decode Dynamo