Menu

Isotonic Calibration Cannot Change Your AUC by a Single Digit. That Is the Point, and Also the Catch.
📰

Isotonic Calibration Cannot Change Your AUC by a Single Digit. That Is the Point, and Also the Catch.

DEV Community: datascience·Devanshu Biswas·13 days ago
#jvKo0r0K
#devtodatascience#science#devto#dev#squash
Reading 0:00
15s threshold

Isotonic regression answers one question: of all non-decreasing sequences, which is closest to this data in least squares? Not approximately — exactly , in one left-to-right pass.

👉 Live, runs in your browser: https://dev48.in/ml/day82-isotonic-regression.html

Its main job in practice is calibration : turning scores that rank well into numbers that mean "probability".

It cannot touch your ranking metrics

Calibration is a monotone map. A monotone map cannot reorder anything. So AUC is untouched by construction :

scores AUC Brier, raw Brier, calibrated improvement

squash 1.00 0.8186 0.1729 0.1739 −0.6%

squash 0.60 0.8186 0.1844 0.1739 5.7%

squash 0.35 0.8186 0.2047 0.1739 15.1%

squash 0.20 0.8186 0.2217 0.1739 21.6%

squash 0.10 0.8186 0.2351 0.1739 26.0%

0.8186 in every row , to four decimals, while Brier moves 26%. The AUC column is not a coincidence to be checked, it is a theorem to be relied on.

This cuts both ways.…

Continue reading — create a free account

Join HashtagPLUS to read full articles, follow hashtags, vote, and join the conversation.