MemT-1.0: How Baking Memory Into the Model Itself Cuts KV Cache Storage by 66x

Every time a large language model processes a long conversation, it pays a tax. The key-value (KV) cache — the data structure that stores intermediate attention states for every past token — grows linearly with context length. At 128,000 tokens, a single request from a model like Llama 3 70B can consume roughly 40 GB of GPU memory just for the cache, competing directly with model weights for scarce high-bandwidth memory. Scale that to a million tokens and the numbers become impractical for most deployments.

The standard responses to this problem — quantizing the cache, compressing it with grouped-query attention, or offloading it to CPU — are engineering patches on a structural issue.…