One step size for every dial

An AI learns by turning dials. A network has millions of them, its weights, and training nudges every one a little at a time. To know which way to turn a dial, training measures how the mistake changes when that dial moves. That measurement is the dial's push, the gradient. It points downhill, toward a smaller mistake.

Plain gradient descent moves every dial with the same rule:

w ← w − step × push

There is one step size, and every dial shares it. The trouble is that hills differ. Take two dials. Both start at w = −2, and both make the smallest mistake at w = 0.

The top dial sits on a steep, narrow hill: mistake = 25 · w². Its push at the start is 100.

The bottom dial sits on a wide, nearly flat hill: mistake = ½ · w². Its push is 2.

Same position, but on the steep hill the push is fifty times bigger.

Two dials, two hills: the same start, but the steep hill's push is fifty times bigger, and one slider sets the step size for both.

Try a big step size, 0.045.…