Fine-tuning interview questions

Three stages, in order. Pretraining: read a massive pile of raw text predicting the next token — the model absorbs language, facts and reasoning, but only learns to continue text. SFT (supervised fine-tuning): imitate curated (instruction → ideal answer) pairs — it gains the behaviour of actually answering, in the right format. RLHF: tune on human rankings so it matches what people prefer — tone, helpfulness, safety. Knowledge, then behaviour, then taste.

Humans compare several answers and rank them best-to-worst. Those rankings train a reward model — a scorer that predicts how much humans would like a given answer. Then the assistant is optimized to produce answers that score higher on that reward, usually with PPO, a reinforcement-learning method, while a KL-divergence penalty keeps it from drifting too far from the sensible SFT model. Newer pipelines often replace PPO with DPO, which tunes straight from the preference pairs.

Pretraining only teaches "what word comes next," so a base model continues your text — ask it a question and it may write more questions. SFT shows it thousands of demonstrations of good assistant behaviour, so it learns to answer the instruction directly, in the right format and tone, and to refuse unsafe requests politely. The knowledge is unchanged; what SFT adds is the behaviour of being helpful.