The BIG idea
one move, repeated
Give the model some text. It predicts what comes next, adds it on, and repeats — again and again — until a whole answer appears.
this loop is called autoregressive generation
Try it first — you be the modelTHE CAT SAT ON THE
Now here's how it actually works
It thinks in PIECES
tokens, not whole words
Models usually don't work with whole words — they work in small chunks of text called tokens. One word can split into several.
un+believe+able=unbelievable
It isn't LOOKING it up
pattern-completion, not a database
When it says "the capital of France is Paris," it isn't finding a stored fact — it's completing the pattern with the words that usually come next. Right most of the time… yet nothing forces it. That's why it can hallucinate.
fluent by design — which is why it can be confidently wrong
Why it's called "LARGE"
billions of learned dials
Those guesses are shaped by billions of weights — plain numbers, nudged over months until the model's guesses matched real text. Not facts you can read; dials that shape every prediction.
each dot ≈ a huge bundle of learned dials
Skills switch on at SCALE
emergence
Reasoning, following instructions, arithmetic — they barely show up in small models, then appear fairly suddenly once a model is big enough. No one hand-codes them; capability falls out of getting better at the guess.
this sudden jump is called emergent ability
The INTERVIEW answer
say it precisely
"An LLM is a next-token predictor — it generates text autoregressively, predicting a probability distribution over the next token given the ones before it." One sentence — loop, tokens, probabilities.
See it in CODE
the whole loop, ~10 real lines
Everything above is this — text becomes tokens, the model scores every possible next one, you pick the top, append it, and loop. Here it is on a real (tiny) model.
from transformers import AutoModelForCausalLM, AutoTokenizer
import torch
# a small "large language model" and its tokenizer
tok = AutoTokenizer.from_pretrained("gpt2")
model = AutoModelForCausalLM.from_pretrained("gpt2")
ids = tok.encode("The capital of France is", return_tensors="pt") # text → tokens
# the core loop: predict → pick → append → repeat
for _ in range(3):
logits = model(ids).logits # a score for EVERY token
probs = torch.softmax(logits[0, -1], dim=-1) # scores → probabilities
next_id = torch.argmax(probs) # pick the most likely next token
ids = torch.cat([ids, next_id.view(1, 1)], dim=1) # append, then loop
print(tok.decode(ids[0])) # → "…is Paris." — completion, not a lookup
the highlighted lines are the exact loop from the diagram above
The questions that END interviews
tap a card — the full answer zooms in, ready to say out loud
The story of one prediction
Model answer
Your tutor
ask about this card
Hi! I'm your tutor for “What is a Large Language Model?” Ask me anything — e.g. “what's a token?” or “why does it hallucinate?”
Why isn't it just a database?
Because it never stores facts to look up. It learned the patterns of text, so it re-generates the most likely continuation. Usually right — but nothing guarantees it, which is why it can confidently make things up.
A Large Language Model (LLM) is a next-token predictor. That sounds almost too simple to be useful — but stack that one trick on billions of weights and the whole internet's worth of text, and you get something that can write code, answer questions, and hold a conversation. This is the mental model everything else in the course builds on.
▶ watch one prediction
How we got here
a 60-second history
The old way: rules. For decades, getting a computer to "understand" language meant hand-writing grammar rules — rigid, brittle, and slow. LLMs threw that out.
Recurrent Neural Networks (RNNs). The first neural approach read text one word at a time, carrying a little memory of what came before. Clever — but they forgot long-range context and were painfully slow to train.
The Transformer (2017). The breakthrough was attention: instead of reading strictly left-to-right, the model looks at every word at once and decides which ones matter for the next guess. Faster to train, far better with long context.
Scale → the LLM. Show that architecture an enormous pile of text with one job — cover the next word and guess it — a trillion times, and to get good at guessing it's forced to absorb grammar, facts, reasoning, and style. The "understanding" is a side effect of getting really, really good at one prediction task.
The five ideas that matter
The whole model is one prediction, repeated
Given the text so far, the model outputs a probability for every possible next token in its vocabulary — not a paragraph, just the odds for the next little piece. It picks one, sticks it on the end, and feeds the whole thing back in to predict the piece after that. This loop is called autoregressive generation. Every impressive thing a Large Language Model does is this same step run hundreds of times in a row.
"Large" means billions of learned numbers
The "L" is literal. A model holds billions of weights — plain numbers that started random and got nudged, over months of training, until its guesses matched real text. GPT-2 had ~1.5 billion; frontier models have hundreds of billions or more. They aren't sentences or facts you can open and read — they're the dials that shape the prediction. More weights and more text generally means subtler patterns, which is why bigger has kept meaning better.
It's pattern-completion, not a database
The single most important interview point. An LLM does not store facts in a table and look them up. When it says "the capital of France is Paris," it isn't querying a row — it's continuing the text with whatever tokens are most likely given everything it read in training. Usually that lands on the truth because the true answer appeared over and over. But nothing forces it, which is exactly why models can hallucinate: confidently produce fluent text that happens to be false.
#1 interview pointA compression of patterns, not a search engine.
Why "just predict" produces something so capable
Here's the leap that surprises people: to guess the next word well, you sometimes have to actually do the task. To finish "2 + 2 =" you need arithmetic. To finish a mystery's last line you need to track the plot. To complete a function you need to know the language. So "predict the next token" quietly pushes the model to learn real skills — because those skills are what make the prediction accurate. Capability falls out of the objective; nobody hand-coded "answer questions" anywhere.
the twistCapability falls out of the objective.
New skills appear only at scale (emergence)
Some abilities — multi-step reasoning, following instructions, basic math — barely show up in small models and then appear fairly suddenly once a model is big enough and trained on enough data. This is called an emergent ability. You can't always predict which skill will pop out at which size, which is part of why the field keeps scaling up.
emergenceYou can’t predict which skill — or when.
The one analogy to remember
Think of the world's most well-read autocomplete. Your phone suggests the next word from the last few you typed; an LLM does the same trick but has read the entire internet, so its "next word" guess can carry a whole coherent thought. It isn't recalling a stored answer — it's finishing your sentence the way its training says sentences like this tend to go.
The whole thing, in plain words
In one breath: it just predicts the next word-piece, again and again, using billions of learned dials — completing the pattern, never looking anything up.
1
Predict & repeat. Text → tokens → odds for the next one → pick → append → loop.
2
“Large” = billions of weights. Learned from internet-scale text, not stored facts.
3
Pattern, not database. Wonderfully fluent — which is why it can confidently make things up.
4
Skills fall out of it. Guessing well needs real skills; some only switch on at scale.
Deep dive · What is an LLM?
Your tutor
ask about this card
Hi! I'm your tutor for “What is a Large Language Model?” Ask me anything — e.g. “what's a token?” or “why does it hallucinate?”
Why isn't it just a database?
Because it never stores facts to look up. It learned the patterns of text, so it re-generates the most likely continuation. Usually right — but nothing guarantees it, which is why it can confidently make things up.