Cross-entropy, the loss behind almost every classifier, as a landscape and as a bet.

A classifier doesn't just pick an answer. It gives every class a score, and softmax turns those scores into probabilities: cat, dog, bird. To learn, the model needs one number that says how good those probabilities were. For classifiers, that number is almost always the cross-entropy loss.

One number, one log

Cross-entropy looks at a single probability: the one the model gave to the right answer. Call it p. The loss is minus its log:

loss = −log(p)

If the picture is a cat and the model said 67% cat, the loss is −log(0.67) ≈ 0.4. If it said 1% cat, the loss is −log(0.01) ≈ 4.6.

The loss as a landscape

Take a classifier with three classes, and hold the bird score still. Two numbers are left: the cat score and the dog score. For every pair, cross-entropy gives a loss. Make that loss the height, and you get a landscape.

The loss over the cat and dog scores: a landscape.…