Tokenization
not letters. You
pay per token.↘
What does tokenization do to your text, and why is it necessary?
A model only does math over numbers, so it can't read letters. Tokenization is the step that bridges that: it splits your text into small pieces, usually subwords, drawn from a fixed vocabulary, then swaps each piece for its integer ID.
What actually enters the network is that row of IDs, one per token. The letters are already gone by then.
Without tokenization there's no numeric input for the model to compute on.
in strawberry?”
→ It reads whole chunks. The letters
are sealed inside a token.
Why split into word-pieces rather than whole words or single letters?
What does BPE buy you?
Two extremes both fail. A whole-word vocabulary would need every word, typo, name and slang term, which is unworkably large, and it still chokes on any word it never saw.
Single letters keep the list tiny but make sequences painfully long and slow.
BPE (Byte-Pair Encoding) is the middle: it starts from characters and repeatedly merges the most common neighbouring pairs into bigger chunks, so frequent bits like "ing" or "tion" become single tokens while a rare word is built from a few smaller known pieces. The vocabulary stays small, and it can still spell out anything, including a brand-new word.
vocabulary
new words
vocabulary
sequences
characters
pairs
anything
Cost and length limits are counted in tokens.
What surprising consequence does that have for a long word?
quietly splits into 3 to 4 tokens
Rough English rule:
100 tokens ≈ ~75 words
are measured in tokens, never words.
cost even more
This is a small sample.
600+ interview questions live inside Skillumen.
Every concept card carries its own interview questions, graded as you answer.
Free to start.