KV-Aware Serving interview questions

The KV cache is the model's running memory of everything it has read — one entry per token — and it dominates serving because it grows with context and eats GPU memory. Generation splits into two phases around it: prefill reads the whole prompt in one pass to build the initial cache (compute-bound), then decode emits tokens one at a time, each re-reading the growing cache (memory-bandwidth-bound). Because they bottleneck on different resources, the whole serving stack is organized around them.

Prefill is one big matrix-heavy burst over every prompt token, so it saturates compute. Decode does tiny per-step math but drags the whole growing KV cache in and out, so it's limited by memory bandwidth. On a single shared GPU they collide: a user with a 100k-token prompt hogs the compute during prefill and creates head-of-line blocking, stalling everyone else's streaming decode. You get poor utilization and jittery latency because one workload starves the other.

Disaggregation puts prefill on one GPU pool and decode on another, shipping the KV cache between them over a fast interconnect (RDMA/NVLink). That fixes the collision: a giant prompt keeps its prefill on pool A without stalling the token streaming on pool B, so you kill head-of-line blocking. It also lets you tune each pool for its bottleneck — compute-heavy hardware for prefill, memory-bandwidth-heavy for decode — and scale them independently. NVIDIA Dynamo and the CNCF's llm-d orchestrate exactly this at cluster scale over engines like vLLM and SGLang.