Deep Learning, Visually #20

Watch the Short: youtube.com/shorts/yYxpjlvW69s

Why the numbers drift

A deep network is a chain of layers. Each layer takes a weighted sum of the activations from the layer before it, passes that sum through an activation function, and hands the result to the next layer.

Training changes the weights at every step. So every layer keeps seeing inputs that have shifted since the last step: a new average, a new spread. In a deep network, those shifts pile up. With a big learning rate, the numbers can drift so far that they blow up, and training dies.

What batch norm does

Batch normalization adds a reset after each layer.…