In “I sat by the bank,” the setting suggests a riverbank. In “I withdrew cash at the bank,” cash points to a financial institution. The spelling stays the same; the surrounding evidence changes.
Self-attention gives each token a query, a key, and a value. The query for “bank” is compared with the keys for the tokens it can see. The resulting weights determine how much each value contributes. Multiple heads can learn different relationships.
In the original encoder–decoder Transformer, the encoder reads the source. The decoder uses causal masking to keep future target tokens out of view and cross-attention to consult the encoded source. Each new token becomes context for the next prediction. The numerical examples in the film are illustrative.
Video: a visual explanation of attention, masking, and decoding.
