FlatCake: Why Tiered KV Cache Storage Is Holding LLM Serving Back

When you send a long prompt to an LLM API, the service has to store the key-value (KV) cache for your session somewhere. For short conversations, GPU HBM handles it fine. But as context windows stretch to hundreds of thousands of tokens — and agentic workflows keep sessions alive for minutes or hours — the KV cache grows large enough that it has to spill off the GPU entirely.

The standard answer has been a tiered hierarchy: GPU HBM → host DRAM → NVMe SSD. DRAM acts as a warm buffer, catching recently evicted blocks before they land on slower SSD. It's the same logic as CPU cache hierarchies, applied to LLM serving. And for a long time, it worked well enough.

A new paper from researchers at Tsinghua University and Zhipu AI argues that this assumption is now wrong.…