Short true-or-false quiz on text length in LLMs. Details vary by model and tokenizer, so treat the answers as general behavior and check the one you use.

The same sentence in English and in Portuguese or Japanese uses about the same number of tokens.

One word is one token.

A model’s context window limit applies only to your prompt.

Two different models will report the same token count for the same text.

A long number or an unusual identifier is usually one token.

Answers below.

.

.

.

Answers:

False in general. Tokenizers are trained on text distributions, and many split non-English text into more tokens than a similar English sentence. The size of the gap depends on the tokenizer.

False. Common words are often one token, but rarer words get split into pieces. For English, a rough rule of thumb is that a token is somewhat less than one word. Code, punctuation and numbers behave differently.

False. The window covers input and output together, and some models also count internal reasoning tokens.…