Rotary Position Embeddings: How Rotating Queries and Keys Encodes Relative Position

Transformers have no recurrence and no convolution, so on their own they cannot tell whether a token came first, fifth, or five-hundredth. Early transformers solved this by adding a position vector to each token embedding. That works, but it mixes "what the token means" with "where the token sits" in a single sum, and it forces the model to learn how position should interact with content from data alone.

Rotary Position Embeddings, introduced as RoFormer in 2021, take a different route. Instead of adding position, RoPE rotates the query and key vectors by an angle that depends on their absolute position. The beautiful consequence is that the attention score between two tokens then depends only on their relative distance. Absolute position goes in, relative position comes out.

The core idea in one picture

Imagine each query and key as a vector in a plane.…