Originally published on almoghagbi.com , where the figures are interactive.
By the end of this post, you'll be able to train a tiny tokenizer by hand.
You'll also predict how it splits a word it has never seen. Every natural
language processing (NLP) model reads text as tokens ,
the units it works with. Section 2.4 of
Speech and Language Processing
shows where models like GPT-4o get theirs: byte-pair encoding (BPE). It starts
from single characters and merges the most frequent pair of neighboring tokens,
again and again.
Intuition
Tokens smaller than words
It's tempting to think of a tokenizer as a dictionary: one token per word. Our
running example, a short review, shows why that fails:
the café baker bakes cakes, the café bakes soufflé
If the vocabulary holds exactly these words, a new review that says "baked"
leaves the model with nothing to work with. This is the unknown word
problem.
Make every character a token instead, and nothing is ever unknown.…