Menu

1 / 8

NLP #3: Byte-Pair Encoding

DEV Community: nlp·GradientGym·about 7 hours ago
#qvoKJXgc
#devtonlp#devto#dev#count#merge
Reading 0:00
15s threshold

Originally published on almoghagbi.com , where the figures are interactive.

By the end of this post, you'll be able to train a tiny tokenizer by hand.

You'll also predict how it splits a word it has never seen. Every natural

language processing (NLP) model reads text as tokens ,

the units it works with. Section 2.4 of

Speech and Language Processing

shows where models like GPT-4o get theirs: byte-pair encoding (BPE). It starts

from single characters and merges the most frequent pair of neighboring tokens,

again and again.

Intuition

Tokens smaller than words

It's tempting to think of a tokenizer as a dictionary: one token per word. Our

running example, a short review, shows why that fails:

the café baker bakes cakes, the café bakes soufflé

If the vocabulary holds exactly these words, a new review that says "baked"

leaves the model with nothing to work with. This is the unknown word

problem.

Make every character a token instead, and nothing is ever unknown.…

Continue reading — create a free account

Join HashtagPLUS to read full articles, follow hashtags, vote, and join the conversation.