“The dog hunted the fox.” “The fox hunted the dog.” They are the same words, but the meanings are totally different. You can note the difference at once, but a transformer can’t.
This is the problem which positional embeddings solve. They inform a language model about the position of each token in the sentence. Without them, one sees a prompt merely as a collection of words.
We will explore these notions starting from basic facts. First, we will find out why transformers are unable to understand word order. Then we will look through three important types of positional embeddings: learned, sinusoidal, and RoPE.
Table of contents
Why Do Transformers Need Positional Embeddings?
Approach 1: Learned Absolute Positional Embeddings
Approach 2: Sinusoidal Positional Encoding
Approach 3: Rotary Position Embeddings (RoPE)
Mini Project: Build RoPE and Verify It Works
What Are Modern LLMs Doing Now?
Positional Embeddings at a Glance:
Common Misconceptions About Positional Embeddings
Conclusion
Frequently…