I trained small Next-Concept Modeling (NCM) models from scratch in MLX on TinyStories. They reconstruct held-out stories almost perfectly, but the text they write on their own is a lot worse than text from a plain autoregressive (AR) baseline built from the same transformer blocks. And the part I didn't expect: the training change that made the teacher-forced loss better made the generated text worse. Same result in two generator seeds.
NCM is the method of Kumbong et al. . My project, ncm-mlx , is an independent MLX implementation of it. The models here are my own small training runs, not reproductions of the authors' models. In an earlier post I wrote about running their released checkpoints in MLX. Below I only use their TinyStories checkpoint as a reference point.
The setup
NCM doesn't write one token at a time. A codec turns a 128-token text window into a stack of discrete codes, from one coarse code at the top down to fine detail at the bottom.…