Serving cost for large language models is usually framed as a compute problem: buy more FLOPs, raise utilization, batch harder. That framing has been drifting out of date for a while. Once KV cache reuse enters the picture, the binding constraint on decode throughput is memory bandwidth, and the numbers operators plan against should move with it.

The Compute Bottleneck Assumption

Most serving dashboards lead with GPU utilization and achieved FLOPs. That made sense when prefill dominated wall-clock time and every optimization targeted matrix multiplication. Teams tuned kernel fusion, chased higher tensor-core occupancy, and treated idle compute as the enemy.

Prefill is genuinely compute-heavy. It processes the whole prompt in parallel, so arithmetic intensity stays high and tensor cores earn their keep. Decode is a different animal. It generates one token at a time, and each step must read the entire key-value cache before producing a single output. The math gets thin fast.

This is why a cluster can report healthy utilization while tokens per second stay flat. The scheduler shows busy SMs, but those SMs spend much of their time waiting on memory transactions. Utilization as a headline metric hides the stall.

PagedAttention and the Reuse Turn

vLLM, originally built at UC Berkeley's Sky Computing Lab, introduced PagedAttention as a memory-management scheme for the KV cache. It borrows the operating system's trick of paging: cache blocks are allocated non-contiguously, so memory fragmentation stops wasting capacity and more concurrent sequences fit on a device.

The same block structure makes prefix caching practical. When many requests share a long system prompt or a common document, the server can point them at the same physical blocks instead of recomputing the prefill. That is a real saving. Shared prompt state gets computed once and reused.

The framework is now widely deployed and has been copied in spirit across the serving stack. Continuous batching, quantization, and OpenAI-compatible APIs arrived alongside it. The reuse turn is not experimental; it is the default posture for anyone serving at volume.

Where the Time Actually Goes

Consider a decode step with a long context. The model must attend over every cached key and value, so the read volume scales with context length while the arithmetic per token stays roughly fixed. Arithmetic intensity, the ratio of math to bytes moved, falls as the prefix grows.

The rough accounting is worth doing on a napkin. For a mid-sized model, the KV cache per token is a few tens of kilobytes per layer per sequence once you account for keys and values across attention heads; multiply by layer count and sequence length and a single long session can hold hundreds of megabytes of cache. Every decode step touches all of it. At high concurrency, the aggregate read volume per step runs into the gigabytes, and that is before weights enter the picture.

On an NVIDIA H100, HBM bandwidth is measured in the low terabytes per second, and that ceiling becomes the practical limit. Vendors quote peak figures under ideal conditions; sustained bandwidth in a real serving loop lands lower. The gap between the two is where throughput quietly disappears.

This is the revision worth internalizing: the decode phase was memory-bound before anyone added caching. Reuse did not create the problem. It made the problem larger and harder to see behind a compute-shaped dashboard.

Reuse Makes Bandwidth Pressure Worse

Here is the counterintuitive part. A cache hit removes prefill work, which is compute, and leaves a longer effective prefix in memory, which is bytes. The optimization trades a resource that was often idle for one that is already saturated.

Longer cached prefixes mean larger reads per token. More concurrent sessions multiply that traffic. The GPU sits with free compute while the memory controllers run hot, and the scheduler cannot fix a bandwidth queue by launching more warps.

Batching helps, since one weight read serves many sequences and amortizes the cost. It does not repeal the KV read. Every active sequence still demands its own cache bytes each step, so the traffic grows with concurrency even as the per-token math stays flat.

What This Changes for Operators

Memory capacity planning starts to outrank FLOPS budgeting. The question shifts from how many tensor cores a node has to how many bytes of KV cache it can hold and how fast it can move them. A related piece on this site argued that training budgets stall while maintainers stay on call; serving budgets have a parallel problem, where the visible line item is not the one that binds.

Cache eviction policy becomes a cost lever. Aggressive eviction frees capacity for new sessions but forces recomputation, which pushes work back to compute. Generous retention protects hit rates and enlarges reads. There is no free setting, only a trade chosen deliberately or by default.

Quantized KV caches trade accuracy for bandwidth relief, shrinking each cached value to fewer bits and cutting read volume proportionally. Interconnect choices matter more than peak TFLOPS too, because multi-node serving moves cache state across links that are far slower than HBM.

The objection worth stating plainly: teams with short prompts and low concurrency will not feel this. If contexts stay under a few thousand tokens and requests arrive one at a time, compute tuning still pays. The bandwidth story bites at long context and high concurrency, which is exactly where the largest serving bills accumulate.

How Reuse Interacts with Batching and Scheduling

Continuous batching is the other half of the modern serving stack, and it compounds the bandwidth story. By admitting new requests into the running batch at every decode step, it keeps the GPU busy and raises throughput. But each admitted sequence brings its own KV cache, and the memory controller must serve all of them each step. The scheduler that maximizes batch size can inadvertently maximize bandwidth pressure.

Operators see this as a plateau: tokens per second rise with batch size up to a point, then flatten or even fall as the memory system saturates. The knee of that curve is the practical operating point, and it moves with context length and cache hit rate. Finding it requires measuring decode throughput across a sweep of batch sizes at realistic sequence lengths, not trusting a single benchmark number.

Prefix caching changes the shape of that curve. High hit rates reduce prefill work and free compute, which might suggest room for larger batches. But the cached prefixes also occupy memory and lengthen the reads per step. The net effect depends on the traffic mix: workloads with many shared long prompts benefit, while diverse short prompts see little gain and some added complexity.

The scheduling interaction also explains a failure mode that looks like a hardware fault. A node can show free memory, low utilization, and still deliver poor tokens per second because the block allocator is thrashing, evicting prefixes between requests that would otherwise hit. Watching hit rate and eviction count side by side separates a genuinely bandwidth-starved node from one whose cache policy is working against it.

Practical Moves for Serving Teams

Instrument KV read volume per request before touching any tuning knob. Log cache bytes touched per decode step alongside tokens per second, so bandwidth pressure shows up as a number rather than a hunch.

Tune prefix cache time-to-live against measured hit rate. Short TTLs save memory and cost recomputation; long TTLs do the reverse. Pick the value from your own traffic mix, not from a default.

Benchmark decode throughput at realistic context lengths. A benchmark run at 512 tokens tells you little about behavior at 8,000, where the read volume per step is far larger and the arithmetic intensity has collapsed.

Evaluate KV quantization on your own evaluation set before buying more GPUs. Lower-bit caches can relieve bandwidth substantially, and the accuracy cost varies by model and task, so measure it rather than assume it.

Treat memory bandwidth as a first-class term in capacity models, next to FLOPS and node count. The same discipline applies elsewhere in infrastructure, as this site noted when rendering pipelines split budgets between runtime and build; the resource you did not budget for is the one that sets the ceiling.