TL;DR

bf16 keeps 8 significant bits. Once a residual-stream value grows to around 600, the gap between neighbouring bf16 numbers is 4, so any single update smaller than 2 is rounded away completely. Matrix multiplies tolerate this. The running sum that carries the residual stream does not, because it adds many small updates to one large number. In a simulation (illustrative, code below) a bf16 residual received between 0.3% and 43% of the drift an fp32 residual received over 4,000 steps, depending on the noise level, and in one setting it stopped moving entirely. The fix used by mainstream training code is cheap: keep the residual path in fp32, keep the matmuls in bf16. If you cannot, stochastic rounding recovers the average in expectation. Whatever you choose, watch residual and gate maxima over time, not just loss.

Why does a bf16 residual stream lose small updates?

A bf16 number has 1 sign bit, 8 exponent bits and 7 stored mantissa bits, which is 8 significant bits including the implicit leading one.…