Meter Cost per Accepted Answer, Not per Call
Last Tuesday a merged pull request fired the "Draft changelog" button in our internal release dashboard, and the model answered in about four seconds with JSON that read fine to a human. It failed one layer later, inside the worker's schema check, because body came back as an array of strings instead of a string. My retry loop then did exactly what I had told it to do: it appended the validation error to the same conversation and asked again, which is a reliable way to make attempt two cost more than attempt one. My dashboard said everything was healthy, because it measured tokens per call.
I want to argue a specific, unhedged position here: the billable unit of an AI feature is the accepted result, and per-call metrics are a form of self-deception that happens to be the easiest thing to build. Every rejected attempt is a small number that dissolves into a large average, so the chart stays flat while the thing you actually pay for quietly gets worse.…