Menu

Why Do AI Models Need High-Dimensional Embeddings?
📰

Why Do AI Models Need High-Dimensional Embeddings?

DEV Community: deeplearning·Karam Sayed·3 days ago
#W2jbaiKb
Reading 0:00
15s threshold

A language model transforms words into context-rich representations that capture the meaning of each word relative to the words around it.

The human mind can visualize up to 3 dimensions intuitively, so let's use a 3-dimensional example to understand why higher dimensions allow richer representations.

Take the sentence: "The dog with that man barked."

Imagine that after the Attention layers process this sentence, the final representation of "dog" lives in a hypothetical 3-dimensional space with axes: Pet (the animal is a pet, hence its association with a man), Barking (the vector points high in this direction, indicating that it is barking), and Attention-seeking (the vector points in this direction, representing a hypothetical association with seeking attention).

Now consider another sentence: "The lion is sleeping."

The final representation of "lion" would be different because its surrounding context and associated properties are different.…

Continue reading — create a free account

Join HashtagPLUS to read full articles, follow hashtags, vote, and join the conversation.