KV-Aware Serving
before compute
runs out.↘
How does the KV cache shape production serving,
and what two generation phases does it split into?
The KV cache is the model's running memory of everything it has read, one entry per token, and it dominates serving because it grows with context and eats GPU memory.
Generation splits into two phases around it. Prefill reads the whole prompt in one pass to build the initial cache, and it is compute-bound.
Then decode emits tokens one at a time, each re-reading the growing cache, and it is memory-bandwidth-bound. Because they bottleneck on different resources, the whole serving stack is organized around them.
on one GPU?”
→ A long prompt's prefill stalls
everyone else's decode stream.
How do prefill and decode differ in what they bottleneck on,
and why does mixing them on one GPU cause problems?
Prefill is one big matrix-heavy burst over every prompt token, so it saturates compute. Decode does tiny per-step math but drags the whole growing KV cache in and out, so it's limited by memory bandwidth.
On a single shared GPU they collide: a user with a 100k-token prompt hogs the compute during prefill and creates head-of-line blocking, stalling everyone else's streaming decode. You get poor utilization and jittery latency because one workload starves the other.
matrix burst
token at once
compute
math
in and out
bound
What problem does disaggregation solve by running prefill and decode
on separate GPU pools linked by a fast connection?
Disaggregation puts prefill on one GPU pool and decode on another, shipping the KV cache between them over a fast interconnect like RDMA or NVLink. That fixes the collision: a giant prompt keeps its prefill on pool A without stalling the token streaming on pool B, so you kill head-of-line blocking.
It also lets you tune each pool for its bottleneck: compute-heavy hardware for prefill and memory-bandwidth-heavy hardware for decode, scaled independently. NVIDIA Dynamo and the CNCF's llm-d orchestrate exactly this at cluster scale over engines like vLLM and SGLang.
This is a small sample.
600+ interview questions live inside Skillumen.
Every concept card carries its own interview questions, graded as you answer.
Free to start.