Menu

My Gradient Clipping Was Never Clipping. It Was Secretly Multiplying My Weight Decay
📰

My Gradient Clipping Was Never Clipping. It Was Secretly Multiplying My Weight Decay

DEV Community: pytorch·b1ackjack12·11 days ago
#99DLnCHo
#devtopytorch#devto#dev#clipping#decay
Reading 0:00
15s threshold

Yesterday's experiment ended with a clean verdict: LR warmup is insurance — free at the tuned learning rate, a full payout at an overheated one. Gradient clipping has the exact same folklore job description. It's the other knob people ship "for stability," the one every training-loop template includes with max_norm=1.0 and no comment. If warmup and clipping are both insurance against early-training violence, they should produce the same 2x2 table. So: same harness, same seeds, Adam at lr 1e-3 and 3e-3, with and without clip_grad_norm_(1.0) , plus an aggressive 0.3 arm, five paired seeds each.

The table that refused to rhyme

arm accuracy (5 seeds) paired Δ t wins

lr 1e-3 89.64 ±0.12 — — —

lr 1e-3 + clip 1.0 89.01 ±0.21 −0.62 −6.4 0/5

lr 3e-3 86.98 ±0.34 — — —

lr 3e-3 + clip 1.0 83.55 ±1.05 −3.43 −10.6 0/5

lr 3e-3 + clip 0.3 73.13 ±1.04 −13.85 −24.5 0/5

Warmup's table, inverted on every row.…

Continue reading — create a free account

Join HashtagPLUS to read full articles, follow hashtags, vote, and join the conversation.