Menu

#AdamW

1 post

Feed
1 of 1 post
Weight Decay 0.05: Catastrophe in Adam, Optimum in AdamW
📰
0Read More

Weight Decay 0.05: Catastrophe in Adam, Optimum in AdamW

DEV Community: pytorch·b1ackjack12·13 days ago
#geaZvlkd
#devtopytorch#devto#dev#adam#adamw

The dose-response series reaches the knob I've been inheriting since September: weight_decay=5e-4, copied from SGD folklore into Adam

15s