September's EMA experiment measured exactly two decay values on the SGD recipe and left a dangling thread: 0.999 catastrophically bad, 0.99 cosmetically good, nothing in between and nothing beyond. Then the stacking post carried decay 0.99 onto the Adam recipe without re-measuring it — an inherited number of precisely the kind this blog keeps catching in other people's configs. The fix is the same one that worked for label smoothing : stop defending a point and sweep the curve. Six decays — 0.9, 0.97, 0.99, 0.995, 0.999, 0.9999 — on the tuned Adam recipe, ten seeds.

The instrument deserves a sentence, because it's the best part. EMA's audit trick from September scales: all six shadows ride inside the same training run, updated from the same weight trajectory after every optimizer step. One run yields seven evaluations — raw plus six decays — so every comparison below is paired at the strongest possible level: same seed, same batches, same trajectory, different averaging window.…