Menu

1 / 5

Why train on a few examples at a time?

DEV Community: deeplearning·Syed Abdul Hanan·15 days ago
#yVjr0qcF
#devtodeeplearning#education#devto#dev#batch
Reading 0:00
15s threshold

Stochastic and mini-batch gradient descent: why every real training loop takes small, noisy steps.

Gradient descent, as we wrote it earlier in this series, has one quiet assumption. To take a single step, it looks at every training example. That is fine for four examples. It is not fine for a million.

Real training never does it that way. It takes a small random handful of examples, a mini-batch, and steps on that alone. This article shows why that works, what it costs, and how it looks in PyTorch. The numbers come from a tiny worked example and from the CS878 Deep Learning slides (lectures 07-08).

The gradient is an average

The loss of a model is an average over the training examples: L = (1/N) Σ ℓᵢ, where ℓᵢ is the loss on example i.…

Continue reading — create a free account

Join HashtagPLUS to read full articles, follow hashtags, vote, and join the conversation.