Tokenization: BPE from Scratch, Vocab Size, and What Tokenizers Hide

Before a language model reads a single word, a tokenizer has already decided how that word gets cut up. The model never sees letters. It never sees words either. It sees token IDs, numbers drawn from a fixed list, and everything the model can and cannot do starts with that list. We built byte pair encoding from scratch, trained it on 83,159 real words from our own published articles, and measured it against OpenAI's real tokenizers. The headline numbers: the same sentence costs 16 tokens in English and 60 in Urdu, a vocabulary stops paying for itself after about 5,000 merges, and the number 2026 is two tokens because the tokenizer chops digits into groups of three.…