Menu

1 / 4

How an LLM Tokenizer Actually Works, in 71 Lines of JavaScript

JavaScript in Plain English - Medium·Ferid Brković·3 days ago
#y25JZTig
Reading 0:00
15s threshold

Member-only story

How an LLM Tokenizer Actually Works, in 71 Lines of JavaScript

Build byte-pair encoding from scratch, train it on 225 words, and see why a short name can cost more tokens than a sentence.

Ferid Brković

8 min read3 hours ago

Press enter or click to view image in full size

I built a tokenizer for this article and fed it two strings. the encoding is working came back as 8 tokens. Привет , a single six-letter word, came back as 12.

Same code, same rules, no special cases. The short word cost more because the tokenizer had never seen anything like it, and that one fact explains most of what is confusing about token counts, context limits and API bills.

The tokenizer is 71 lines of plain JavaScript with no dependencies. By the end of this article you will have read all of it, watched it learn from 225 words of text, and know what the real ones inside GPT-style models add on top.

What does the model see instead of your prompt?

Not words, and not letters.…

Continue reading — create a free account

Join HashtagPLUS to read full articles, follow hashtags, vote, and join the conversation.