Menu

LLM Tokens, Measured in C#: Why AI Can't Count the R's in Strawberry
📰

LLM Tokens, Measured in C#: Why AI Can't Count the R's in Strawberry

DEV Community: dotnet·VLAD·17 days ago
#VIg9fX0S
#devtodotnet#devto#dev#tokens#token
Reading 0:00
15s threshold

In August 2024, TechCrunch asked a simple question : how many R's are in "strawberry"? According to GPT-4o and Claude at the time, two. The answer is three.

The model isn't bad at counting. It never saw the letters. It saw tokens, and every token is just a number.

Every token count in this article was measured by me on October 30, 2026, with Microsoft's own .NET library, Microsoft.ML.Tokenizers 2.0.1, using the tokenizer for gpt-5 (the same o200k_base encoding GPT-4o uses). The program is at the end, so you can check them.

What the model actually gets

Before your prompt reaches the model, a tokenizer cuts it into pieces, called tokens, and turns each piece into an ID from a fixed list.

How many r's are in strawberry? becomes 8 tokens:

Token How many r 's are in strawberry ?

ID 5299 1991 428 885 553 306 101830 30

Look at strawberry , with the space in front. It's one token : number 101830.…

Continue reading — create a free account

Join HashtagPLUS to read full articles, follow hashtags, vote, and join the conversation.