Every recipe on this blog since the optimizer face-off has carried weight_decay=5e-4 β a number I originally picked for SGD because every CIFAR repo on the internet uses it, then ported to Adam when Adam won, with exactly zero measurement of whether it belongs there. That's the precise shape of inherited config this dose-response series exists to audit. So: Adam at four doses spanning 2.5 orders of magnitude plus a zero control, AdamW at the matching doses, five seeds each on the usual 10-epoch CIFAR ResNet18 β forty-five runs, with each optimizer's arms compared against its own wd=0 baseline on identical seeds.
The reason to sweep both optimizers is that "weight decay" is secretly two different mechanisms sharing a keyword argument. In Adam, the classic implementation adds wd * w to the gradient , which then flows through the adaptive machinery and gets divided by the running RMS of the gradients.β¦