Decoder-Only Transformers

In this article, we will explore decoder-only transformers .

Decoder-only transformers are a specific type of transformer architecture used in systems like ChatGPT.

Masked Self-Attention

Decoder-only transformers use a mechanism called masked self-attention .

Masked self-attention works by measuring how similar each word is to itself and to the words that come before it in the sentence.

For example:

“The pizza came out of the oven and it tasted good.”

When processing the word “pizza” , masked self-attention only considers the preceding word “The” .

Key Difference

Unlike standard self-attention, masked self-attention does not allow a word to look at future words .…