A language model transforms words into context-rich representations that capture the meaning of each word relative to the words around it.
The human mind can visualize up to 3 dimensions intuitively, so let's use a 3-dimensional example to understand why higher dimensions allow richer representations.
Take the sentence: "The dog with that man barked."
Imagine that after the Attention layers process this sentence, the final representation of "dog" lives in a hypothetical 3-dimensional space with axes: Pet (the animal is a pet, hence its association with a man), Barking (the vector points high in this direction, indicating that it is barking), and Attention-seeking (the vector points in this direction, representing a hypothetical association with seeking attention).
Now consider another sentence: "The lion is sleeping."
The final representation of "lion" would be different because its surrounding context and associated properties are different.…