Menu

Are your model's probabilities calibrated? Expected calibration error in NumPy
📰

Are your model's probabilities calibrated? Expected calibration error in NumPy

DEV Community: datascience·geodesic partners·about 24 hours ago
#Sik6WdcJ
Reading 0:00
15s threshold

There was a comment on Hacker News this week saying LLM token probabilities are badly calibrated for making decisions. I think that's right, and it's also something you can measure yourself in about fifteen lines, so here's how I'd do it.

The usual number is expected calibration error. You take a set of predictions where you already know the outcome and put the stated probabilities into m equal-width bins. In each bin you compare the average stated probability with the fraction of events that actually happened, weight that gap by the share of all predictions that fell in the bin, and add it up.

import numpy as np

def ece ( p , y , m = 10 ) : p , y = np . asarray ( p , float ) , np . asarray ( y , float ) b = np . minimum ( ( p * m ) . astype ( int ) , m - 1 ) # p = 1.0 goes in the last bin total = 0.0 for k in range ( m ) : s = b == k if s . any ( ) : # empty bins carry no weight total += s . mean ( ) * abs ( y [ s ] . mean ( ) - p [ s ] .…

Continue reading — create a free account

Join HashtagPLUS to read full articles, follow hashtags, vote, and join the conversation.