Menu

1 / 2

Understanding Multi-Head Attention in Transformers

DEV Community·Rijul Rajesh·7 months ago
#NYl9St9J
#ai#machinelearning#software#coding#attention
Reading 0:00
15s threshold

Self-attention already helps a transformer understand relationships between words using Query, Key, and Value. But there’s a problem.

One attention mechanism usually ends up focusing on a limited kind of relationship at a time.

Language doesn’t work like that. A sentence can have structure, meaning, and long-range links all at once.

That’s why transformers use multi-head attention .

What happens in multi-head attention

Instead of doing attention once, the model does it multiple times in parallel.

Each run is called a head, and each head has its own learned weights for Query, Key, and Value.

So every head looks at the same sentence, but in its own way.

How it flows

The input embeddings are first prepared

They are split into multiple heads using linear projections

Each head runs its own self-attention

Each head produces its own output

All outputs are joined back together

A final layer mixes them into one result

Why this works better compared to previous approach

Different heads naturally pick…

Continue reading — create a free account

Join HashtagPLUS to read full articles, follow hashtags, vote, and join the conversation.