Menu

Counting tokens exactly, and why the same sentence costs more in Hindi
📰

Counting tokens exactly, and why the same sentence costs more in Hindi

DEV Community: python·Createchz·5 days ago
#7kSpeQOp
#devtopython#devto#dev#tokens#token
Reading 0:00
15s threshold

Createchz

Posted on Nov 14

Edited on Nov 14

AI-assisted

Every LLM price, context limit and output cap is measured in tokens, so it pays to count them properly. Here is the short version, with the numbers I measured.

What a token is

A token is a chunk of text: a short word, part of a longer word, a few digits or a symbol. OpenAI's rule of thumb is about 4 characters or 0.75 words per token in English ( OpenAI API concepts ). With tiktoken's o200k_base , "tokenization" is 2 tokens, "2026" splits into 202 + 6 , and the German compound "Straßenbahnhaltestelle" splits into 7. The vocabulary comes from byte-pair encoding, applied to subwords as in Sennrich, Haddow and Birch (2015) .

Count exactly (OpenAI)

import tiktoken enc = tiktoken . get_encoding ( "o200k_base" ) print ( len ( enc . encode ( "Hello world" ) ) ) # 2

Enter fullscreen mode

Exit fullscreen mode

tiktoken is OpenAI's open-source tokenizer.…

Continue reading — create a free account

Join HashtagPLUS to read full articles, follow hashtags, vote, and join the conversation.