Stochastic and mini-batch gradient descent: why every real training loop takes small, noisy steps.
Gradient descent, as we wrote it earlier in this series, has one quiet assumption. To take a single step, it looks at every training example. That is fine for four examples. It is not fine for a million.
Real training never does it that way. It takes a small random handful of examples, a mini-batch, and steps on that alone. This article shows why that works, what it costs, and how it looks in PyTorch. The numbers come from a tiny worked example and from the CS878 Deep Learning slides (lectures 07-08).
The gradient is an average
The loss of a model is an average over the training examples: L = (1/N) Σ ℓᵢ, where ℓᵢ is the loss on example i.…