Createchz
Posted on Nov 14
Edited on Nov 14
AI-assisted
Every LLM price, context limit and output cap is measured in tokens, so it pays to count them properly. Here is the short version, with the numbers I measured.
What a token is
A token is a chunk of text: a short word, part of a longer word, a few digits or a symbol. OpenAI's rule of thumb is about 4 characters or 0.75 words per token in English ( OpenAI API concepts ). With tiktoken's o200k_base , "tokenization" is 2 tokens, "2026" splits into 202 + 6 , and the German compound "Straßenbahnhaltestelle" splits into 7. The vocabulary comes from byte-pair encoding, applied to subwords as in Sennrich, Haddow and Birch (2015) .
Count exactly (OpenAI)
import tiktoken enc = tiktoken . get_encoding ( "o200k_base" ) print ( len ( enc . encode ( "Hello world" ) ) ) # 2
Enter fullscreen mode
Exit fullscreen mode
tiktoken is OpenAI's open-source tokenizer.…