Tokenization interview questions

A model only does math over numbers — it can't read letters. Tokenization is the step that bridges that: it splits your text into small pieces (usually subwords) drawn from a fixed vocabulary, then swaps each piece for its integer ID. What actually enters the network is that row of IDs, one per token — the letters are already gone. Without it there's no numeric input for the model to compute on.

Two extremes both fail. A whole-word vocabulary would need every word, typo, name and slang term — unworkably large, and it still chokes on any word it never saw. Single letters keep the list tiny but make sequences painfully long and slow. BPE (Byte-Pair Encoding) is the middle: it starts from characters and repeatedly merges the most common neighbouring pairs into bigger chunks, so frequent bits like "ing" or "tion" become single tokens while a rare word is built from a few smaller known pieces. Small vocabulary, but it can still spell out anything — even a brand-new word.

A long or unusual word doesn't cost "one word" — it can quietly split into three or four tokens, and both your bill and your context window are measured in tokens, not words. So a prompt that looks short by word count can be far larger in tokens (rough English rule: 100 tokens ≈ 75 words). Code, math, emoji, long URLs and non-English text tokenize into many more pieces than plain English, so they cost more and fill the window faster for the same apparent length.