Ask what a token embedding is and you'll usually get something like "it captures the meaning of a word as a vector." That sounds profound and tells you nothing you can act on.

Here is the mechanical version: an embedding is a row of a matrix, looked up by an integer. That's it. Everything interesting comes from what training does to those rows, not from the lookup.

If you know what a vector is, you already have the tools to follow the whole thing.

From text to an integer

Before anything mathematical happens, the tokeniser turns your text into integers:

"data science" → ["data", " science"] → [1223, 8622]

Enter fullscreen mode

Exit fullscreen mode

Those numbers aren't meaningful. Token 8622 isn't twice as anything as token 4311.…