Embeddings interview questions

An embedding replaces a token's ID with a learned dense vector — a long list of numbers, say 384 or 4096 of them. A raw ID like 4127 is just a name-tag: it carries no meaning and 893 isn't "less than" 4127 in any useful sense. The embedding packs meaning into the numbers themselves, arranged so similar meanings sit close together. That's what lets the model do math on ideas instead of matching exact spellings.

It means the vectors are placed so that words used in similar ways end up as nearby points — "cat", "dog", "puppy" cluster together, far from "car". You measure "how close" with cosine similarity: roughly, do the two vectors point the same direction? A score near 1 = very similar, near 0 = unrelated. This is why a model can treat synonyms almost interchangeably without ever being told they're synonyms.

Directions in the space are meaningful, not just distances. The arrow from "man" to "woman" turns out to be about the same as the arrow from "king" to "queen" — a consistent "gender" direction. So if you take the vector for king, subtract man and add woman, you walk along that direction and land almost exactly on queen. It shows relationships are encoded as parallel directions, which is why models generalize instead of memorizing each word in isolation.