Menu

Lower loss, worse stories: small NCMs trained on TinyStories
📰

Lower loss, worse stories: small NCMs trained on TinyStories

DEV Community: nlp·Jan·13 days ago
#fxlsIz7g
#devtonlp#devto#dev#training#generator
Reading 0:00
15s threshold

I trained small Next-Concept Modeling (NCM) models from scratch in MLX on TinyStories. They reconstruct held-out stories almost perfectly, but the text they write on their own is a lot worse than text from a plain autoregressive (AR) baseline built from the same transformer blocks. And the part I didn't expect: the training change that made the teacher-forced loss better made the generated text worse. Same result in two generator seeds.

NCM is the method of Kumbong et al. . My project, ncm-mlx , is an independent MLX implementation of it. The models here are my own small training runs, not reproductions of the authors' models. In an earlier post I wrote about running their released checkpoints in MLX. Below I only use their TinyStories checkpoint as a reference point.

The setup

NCM doesn't write one token at a time. A codec turns a 128-token text window into a stack of discrete codes, from one coarse code at the top down to fine detail at the bottom.…

Continue reading — create a free account

Join HashtagPLUS to read full articles, follow hashtags, vote, and join the conversation.