Why static foundation models hit a wall on unfamiliar tasks—and how replacing fixed hidden states with self-updating neural weights enables continuous adaptation at inference time.
For over a decade, modern deep learning has enforced a strict wall between training and inference. During pre-training, massive GPU clusters spend months updating billions of weights via gradient descent. Once deployment begins, those weights are frozen in stone.
At inference time, a deployed model can only adapt to new information by stuffing tokens into its context window. The moment that session ends—or the context window rolls over—everything the model "learned" during the interaction vanishes.…