There was a comment on Hacker News this week saying LLM token probabilities are badly calibrated for making decisions. I think that's right, and it's also something you can measure yourself in about fifteen lines, so here's how I'd do it.
The usual number is expected calibration error. You take a set of predictions where you already know the outcome and put the stated probabilities into m equal-width bins. In each bin you compare the average stated probability with the fraction of events that actually happened, weight that gap by the share of all predictions that fell in the bin, and add it up.
import numpy as np
def ece ( p , y , m = 10 ) : p , y = np . asarray ( p , float ) , np . asarray ( y , float ) b = np . minimum ( ( p * m ) . astype ( int ) , m - 1 ) # p = 1.0 goes in the last bin total = 0.0 for k in range ( m ) : s = b == k if s . any ( ) : # empty bins carry no weight total += s . mean ( ) * abs ( y [ s ] . mean ( ) - p [ s ] .…