One 128K-token conversation with a 70B model needs 40 GiB of GPU memory — not for the weights, for the memory of the conversation itself . An 8B model on the same GPU fits 56 such conversations at once; the 70B model in FP8 fits zero. Same GPU, same 8K-token chats, and the difference is not speed. It is memory. Every LLM serving decision you will ever make is downstream of this one fact: the KV cache, not the FLOPs, caps your concurrency.
The fix has a name you have heard a hundred times and a mechanism most people never actually work through: PagedAttention. It treats the KV cache like an operating system treats virtual memory — fixed-size pages, allocated on demand, freed the moment a request ends. The 2023 paper behind it reports 2–4x higher throughput at the same latency, with memory waste falling from 60–80% to under 4%.
And the ground just shifted under it. On November 5, 2026 — five days ago — vLLM merged PR #63231, changing the default block size from 16 tokens to 64.…