Valenx

Posted on Nov 11

AI-assisted

What the interviewer is testing

Whether you can explain the query-key-value computation, why self-attention improves long-range token interactions, the quadratic cost, and the reason for scaling before softmax.

Answer framework

Represent each token with three learned projections: a query, a key and a value.

Compute compatibility scores with the query-key dot product, scale them by 1/sqrt(d_k), and apply softmax to obtain attention weights.

Produce each token's output as a weighted sum of the value vectors using those attention weights.

Allow each token to retrieve information from any position permitted by the attention mask in one layer, avoiding a recurrent bottleneck while supporting parallel sequence processing during training.

Recognize the trade-off: full self-attention has O(n^2) compute and an attention matrix that also scales quadratically with sequence length.

Sample answer

I explain self-attention as a way for the model to decide which other tokens…