Gradient descent, the walk every neural network takes while it learns, step by step with real numbers.

Imagine standing in the hills with a blindfold on, trying to reach the bottom of the valley. You can't see the valley or the path. All you have is what you feel under your feet: which way the ground tilts, and how steeply.

That's enough. If the ground tilts down to your left, step left, then feel again. Repeat until the ground is flat. That loop is gradient descent, and it's how almost every neural network is trained.

The ground is the loss

A network in training is in the same position. Its loss, one number that says how wrong its answers are, depends on its weights: change a weight and the loss goes up or down.

Picture a network with just two weights. For every pair of weights you can compute the loss, so you can draw it as a height over a flat map of the two weights. That gives a landscape. Where you stand on the map is the current weights. The height under your feet is the current loss.…