Menu

Warmup Did Nothing for My Learning Rate — Then It Fully Rescued the Wrong One
📰

Warmup Did Nothing for My Learning Rate — Then It Fully Rescued the Wrong One

DEV Community: pytorch·b1ackjack12·11 days ago
#p60Jc4a0
#devtopytorch#devto#dev#warmup#rate
Reading 0:00
15s threshold

Learning-rate warmup has the strangest status of any training-recipe fragment: universally shipped, rarely justified. It earned its reputation in transformer training, where skipping it genuinely detonates runs, and from there it diffused into every config file on the internet — including CNN recipes at small scale, where nobody seems to have asked whether it does anything. My own schedule face-off never tested it; the dose-response series suggests the right question isn't "is warmup good?" but "under what condition does the knob engage?". So, a 2x2: Adam at the tuned lr 1e-3 and at an overheated 3e-3, each with and without one epoch of linear warmup, cosine after, five paired seeds each.

The 3e-3 arm isn't arbitrary. Back in the optimizer face-off's screening , Adam at 3e-3 was one of the single-seed losers that got eliminated — a learning rate I know is wrong for this recipe.…

Continue reading — create a free account

Join HashtagPLUS to read full articles, follow hashtags, vote, and join the conversation.