A deep dive into memory fragmentation, paged memory management, and why PagedAttention can deliver up to 24× higher throughput than conventional KV cache implementations.

Every token you generate during LLM inference silently eats GPU memory. With traditional KV caching, a significant portion of that memory is wasted — never used, never reclaimed. vLLM’s PagedAttention changed that by borrowing a decades-old idea from operating systems. Here’s exactly how it works and why it matters.

Table of Contents

What Is a KV Cache and Why Does It Exist?

The Problem: Traditional KV Cache and Memory Fragmentation

Inspiration from OS Virtual Memory — The vLLM Insight

PagedAttention: How It Works

Memory Fragmentation: Before vs After

Throughput Gains: Numbers and Benchmarks

Trade-offs and Limitations

Who Should Care About This?

Key Takeaways

Section 1 — What Is a KV Cache and Why Does It Exist?

What to cover:

During autoregressive decoding, each new token attends to all previous tokens

Without caching:…