Transformers did not get slow because attention needs too many multiplications. They got slow because attention moves too much data. FlashAttention is a lesson in that distinction. It does not change what attention computes. It changes where the intermediate results live, how often they are read and written, and in what order the work happens. The result is the same attention output, computed with far less traffic to slow memory, and therefore much faster in practice, especially for long sequences.
This article explains the memory wall behind standard attention, the online softmax trick that makes tiling possible, the forward and backward passes of FlashAttention, and the limits that remain.…