Fine-tuning
behaviour, then
taste.↘
Name the three stages of building a model and what each one teaches.
Three stages, in order. Pretraining: the model reads a massive pile of raw text predicting the next token.
It absorbs language, facts and reasoning, but only learns to continue text.
SFT (supervised fine-tuning): the model imitates curated pairs of an instruction and its ideal answer, and gains the behaviour of actually answering, in the right format. RLHF: tuning on human rankings so it matches what people prefer in tone, helpfulness and safety.
Knowledge, then behaviour, then taste.
language, facts, reasoning
the behaviour of answering
what people prefer
to RLHF?”
→ No. You can't rank answers until the
model can produce reasonable ones.
It's a ladder.
In RLHF, how do human rankings get turned into something that tunes the model,
and what algorithm is often used?
Humans compare several answers and rank them best-to-worst. Those rankings train a reward model, a scorer that predicts how much humans would like a given answer.
Then the assistant is optimized to produce answers that score higher on that reward, usually with PPO, a reinforcement-learning method, while a KL-divergence penalty keeps it from drifting too far from the sensible SFT model. Newer pipelines often replace PPO with DPO, which tunes straight from the preference pairs.
answers
model (scorer)
a higher score
don't drift far
from SFT
straight from
the pairs
What behaviour does SFT add that pretraining alone leaves missing?
Pretraining only teaches "what word comes next," so a base model continues your text. Ask it a question and it may write more questions.
SFT shows it thousands of demonstrations of good assistant behaviour, so it learns to answer the instruction directly, in the right format and tone, and to refuse unsafe requests politely. The knowledge is unchanged.
What SFT adds is the behaviour of being helpful.
a question
more questions
a question
right format + tone
This is a small sample.
600+ interview questions live inside Skillumen.
Every concept card carries its own interview questions, graded as you answer.
Free to start.