Blame travels backward

When a neural network makes a mistake, training works out how much each dial (each weight) was to blame. That blame starts at the answer and travels backward, layer by layer, so every dial knows which way to turn. This is backpropagation.

On the way back, the blame passes through each layer's bend, the activation function. At every layer it gets multiplied by that bend's slope. This is the chain rule: one slope per layer, all multiplied together. In a deep network, that long chain of multiplications can shrink to almost nothing, or grow out of control.

A quarter at a time

Older networks used an S-shaped bend, the sigmoid: σ(s) = 1 / (1 + e⁻ˢ). Its slope is σ(s)(1 − σ(s)). That slope is steepest in the middle, at s = 0, where it is exactly 0.25. Everywhere else it is smaller, and far from the middle it is almost zero (the lecture calls these neurons saturated: they "kill the gradient flow").

So each sigmoid layer passes back at most a quarter of the blame.…