Autograd, step by step: PyTorch records the forward pass as a graph, then replays it backward.

In the last article we did backpropagation by hand. We sent the error back through a tiny network, one node at a time, and found each weight's gradient. It worked, but nobody does that for a real network with millions of weights.

In PyTorch you write the forward pass, call L.backward() , and every gradient is there. This article shows what happens in between. The engine that does it is called autograd , short for automatic differentiation.

We'll use the same network and the same numbers as last time, from the CS878 Deep Learning slides.

The forward pass builds the graph

Here is the network in PyTorch:

x = torch . tensor ( 1.0 ) w1 = torch . tensor ( 1 / 3 , requires_grad = True ) w2 = torch . tensor ( 2.0 , requires_grad = True ) y = torch . tensor ( 0.0 ) z = x * w1 s = z .…