Rolling down a bowl with noisy pushes

Training an AI is like rolling a ball down a bowl. The ball's position is the AI's dials (its weights), the height is how many mistakes it makes, and the bottom means the fewest mistakes. Each training step moves the ball a little downhill:

w ← w − step size × push

The step size is the learning rate and the push is the gradient. But each push comes from just a few examples (a mini-batch), not all the data, so it always points a little off the true downhill direction.

The bowl in the Short: mistake = ½ · (w1² + 4 · w2²), a long oval with its bottom at (0, 0). The ball starts at (−4, 1.5), 4.27 from the bottom. Each push is the true slope (w1, 4 · w2) plus 0.6 × a normal random number in each direction, from numpy's default_rng(2252). All three runs below use the same noise and take 60 steps.

Big steps bounce, small steps crawl

With a big step size, 0.45, the ball is about 0.2 from the bottom by step 6.…