MemT‑1.0 shatters the long‑standing belief that extending context length inevitably inflates memory consumption, delivering a two‑order‑of‑magnitude compression (~66×) with only negligible accuracy loss. The paper’s Mixture‑of‑Storage architecture packs token histories into compact slots, turning what used to be a linear KV cache into a sublinear memory footprint.
Before MemT‑1.0, transformer models relied on a full key‑value cache that grew linearly with each additional token, forcing practitioners either to truncate context or to augment the model with external retrieval systems that still maintained a complete cache for attention. Those workarounds preserved raw capacity but did not address the fundamental scaling bottleneck.
Mixture‑of‑Storage cuts persistent storage at a 128K‑token history by roughly 66× compared with an uncompressed KV cache, making the entire context fit into a fraction of the original memory budget.…