A ResNet-34 trained on CIFAR-100 is measurably overconfident: its top prediction bin claims 99% certainty while landing correctly only 94% of the time. Temperature scaling, a single learned number, cuts that gap by nearly two-thirds without changing a single prediction.

Key takeaways

A registered gate said: if Expected Calibration Error (ECE) on the held-out half of the test set came in under 3%, the claim "this model is overconfident" would be dead. The real number is 7.87%, more than double the threshold.

Temperature scaling cut ECE from 7.87% to 2.85%, a 63.7% relative reduction, well past the 30% bar registered to count as a fix.

The gap concentrates where most predictions live: 70.1% of test examples fell in the top confidence bin, claiming 99.11% certainty on average while landing correctly 94.27% of the time, before any correction.

A single scalar, fitted on one held-out half of the data and applied to the other, does this without moving accuracy at all: 80.24% before and after.

What is Expected…