Loss functions turn every mistake into one number. Here's how squared and absolute loss score the same misses very differently.

A neural network can't feel embarrassed when it's wrong. It needs a way to measure its mistakes, so it knows which direction to improve.

That measurement is the loss function. It turns all of a network's errors into a single number, and training is the process of pushing that number down.

One number for "how wrong"

A loss function compares the network's predictions with the right answers and returns one number:

Zero means every prediction is perfect.

Bigger means worse.

That's it. Everything a network learns, it learns by trying to make this number smaller.

A simple example

Say a network draws a line to predict three data points. It doesn't fit perfectly. Its predictions miss the three points by 3, 2, and 1.

How bad is that, in one number?…