Backpropagation, step by step: how one backward pass hands every weight its share of the blame.

When a neural network makes a wrong prediction, training has to fix it. But a network has many weights, and each one played some part in the mistake. Before we can change anything, we need to know how much each weight is to blame.

That's what backpropagation does. After the network makes its prediction (the forward pass), it sends the error back through the network (the backward pass). On the way, every weight picks up a number, its gradient, that says how much the loss would change if that weight moved a little. That number is the weight's share of the blame.

The examples below are the ones from the CS878 Deep Learning lecture slides, with the same numbers.

A tiny graph, forward

Start with something small enough to see all at once:

f(x, y, z) = (x + y) · z

Draw it as a graph with two steps. An add node computes d = x + y, and a multiply node computes f = d · z.

Put in x = 1, y = −3 and z = 4.…