The chain rule, the idea that lets a network see through every one of its layers, with real numbers.

A neural network is a chain. The input goes into the first layer, the first layer's output goes into the second, and so on, until the last layer produces an answer and a loss, one number that says how wrong that answer is.

Training keeps asking one question: if I nudge this weight a little, how much does the loss move? For a weight ten layers down, the nudge has to travel through every layer above it before it reaches the loss. The chain rule tells you how big it is when it arrives.

A network is a chain

Written as math, a ten-layer network is a function inside a function inside a function:

f₁₀( ⋯ f₂( f₁(x) ) ⋯ )

Picture it as a tower of dials on one shaft: the input at the bottom, a dial for each layer above it. Turn the bottom dial a little and every dial above it turns too, but not by the same amount.…