Member-only story
How an LLM Tokenizer Actually Works, in 71 Lines of JavaScript
Build byte-pair encoding from scratch, train it on 225 words, and see why a short name can cost more tokens than a sentence.
Ferid Brković
8 min read3 hours ago
Press enter or click to view image in full size
I built a tokenizer for this article and fed it two strings. the encoding is working came back as 8 tokens. Привет , a single six-letter word, came back as 12.
Same code, same rules, no special cases. The short word cost more because the tokenizer had never seen anything like it, and that one fact explains most of what is confusing about token counts, context limits and API bills.
The tokenizer is 71 lines of plain JavaScript with no dependencies. By the end of this article you will have read all of it, watched it learn from 225 words of text, and know what the real ones inside GPT-style models add on top.
What does the model see instead of your prompt?
Not words, and not letters.…