Question 1

In plain terms, what does self-attention let each token do?

Accepted Answer

Self-attention lets every token glance at all the other tokens in the sequence and pull in meaning from the ones most relevant to it — in a single step. Instead of reading a word in isolation, the model rebuilds each token's vector as a blend of the words it cares about, so “it” can absorb the meaning of “cat” however far back it sits. That context-mixing is the core trick behind the transformer.

Question 2

Walk through the three steps — scoring, softmax, and mixing — that produce a token's new vector.

Accepted Answer

Three steps. Score: compare the token's Query against every token's Key with a dot product, then divide by the square root of the vector size so the numbers don't blow up. Softmax: push those scores through softmax so they become weights that add up to 1 — a clean distribution over “how much do I attend to each word?”. Mix: take a weighted sum of everyone's Values using those weights. The result is a fresh, context-aware vector for the token.

Question 3

What are the Query, Key, and Value roles, and how are they made from a token's vector?

Accepted Answer

Each is a small vector made by multiplying the token's embedding by its own learned matrix (Wq, Wk, Wv). The Query is “what am I looking for?” — the question the token shops with. The Key is “what do I offer?” — its name-tag, matched against other Queries. The Value is “the actual information I'd hand over” if I'm chosen. Queries are matched to Keys to decide the weights; the Values are what actually get blended.

Attention interview questions

In plain terms, what does self-attention let each token do?

Walk through the three steps — scoring, softmax, and mixing — that produce a token's new vector.

What are the Query, Key, and Value roles, and how are they made from a token's vector?

Also asked in interviews

Keep going