Long-context models can now handle hundreds of thousands, even millions of tokens. But I've been wondering if we're focusing too much on recall and not enough on what happens when information changes.

Say a coding agent reads that an API timeout is 30 seconds. Later, the documentation changes it to 5 seconds. Both versions are still in context, but the agent keeps using 30.

It hasn't forgotten anything. It just hasn't updated what it knows.

I've been reading about different memory architectures, and this distinction seems pretty important.

With Transformers and RAG, old and new information can coexist, and the model has to resolve the conflict when reading it. In simple linear attention, associations accumulate in a shared state, so writing something new doesn't necessarily remove what's already there. DeltaNet tries to address this through prediction-error-based updates.…