Attention
at every other.
In one step.↘
In plain terms, what does self-attention let each token do?
Self-attention lets every token glance at all the other tokens in the sequence and pull in meaning from the ones most relevant to it, in a single step. Instead of reading a word in isolation, the model rebuilds each token's vector as a blend of the words it cares about, so “it” can absorb the meaning of “cat” however far back it sits.
That context-mixing is the core trick behind the transformer.
every pair cost?”
→ Work grows with the square of the
length. Double the text and you
roughly quadruple the cost.
Walk through the three steps, scoring, softmax, and mixing,
that produce a token's new vector.
Three steps. Score: compare the token's Query against every token's Key with a dot product, then divide by the square root of the vector size so the numbers don't blow up.
Softmax: push those scores through softmax so they become weights that add up to 1, a clean distribution over how much to attend to each word.
Mix: take a weighted sum of everyone's Values using those weights. The result is a fresh, context-aware vector for the token.
Query · Key, ÷ √(vector size)
scores become weights
that sum to 1
weighted sum of the Values
What are the Query, Key, and Value roles,
and how are they made from a token's vector?
Each is a small vector made by multiplying the token's embedding by its own learned matrix (Wq, Wk, Wv). The Query is what the token is looking for.
The Key is what it offers, the name-tag matched against other tokens' Queries. The Value is the actual information it hands over if it's chosen.
Queries are matched to Keys to decide the weights, and the Values are what actually get blended.
This is a small sample.
600+ interview questions live inside Skillumen.
Every concept card carries its own interview questions, graded as you answer.
Free to start.