Every Transformer-based language model, from BERT to GPT, relies on a mechanism called attention to determine how different tokens in a sequence relate to one another. At the heart of this mechanism are three representations: Query (Q), Key (K), and Value (V).

Let's understand what they are, why we need them, and how they work together to help Transformers process language.

1. What Are Query, Key, and Value (QKV)?

To understand QKV, imagine searching for information in a large collection of documents. You have a question you want answered, each document has information that helps you decide whether it is relevant, and each document contains the actual content you might retrieve.

The attention mechanism follows a conceptually similar process: it determines what information a token should gather from other tokens in the sequence.

Query (Q) — What information am I looking for?

A Query represents what a token is looking for in other tokens.…