You want a model to look over a day of sensor readings and flag anything odd. One reading a minute is 1,440 rows, and you write them out with Python's csv module, the obvious way. That file is 49,723 tokens. The same information fits in 17,287. The other 32,436 tokens are digits that either repeat on every row or mean nothing.

The model reads digits three at a time

You pay per token , a short chunk of text the model reads as one piece. Input tokens are what you send, output tokens are what it writes back, and each has its own price per million. A tokenizer is the program that does the cutting. I used tiktoken, OpenAI's open-source one, with its o200k_base encoding.

For numbers the rule is mechanical: digits are cut into groups of up to three, left to right. 19.94 is three tokens: 19 , . , 94 . 19.944444444444446 is seven: 19 , . , 944 , 444 , 444 , 444 , 446 .

That second number is what Python gives you for (67.9 - 32) * 5 / 9 , a Fahrenheit reading converted to Celsius.…