LLM Interview Questions (2026): The Ones Interviewers Really Ask

It's a next-token predictor. You give it some text, and it outputs a probability distribution over what the next chunk of text is most likely to be, picks one, appends it, and repeats. That's it. Everything else — answering questions, writing code, sounding thoughtful — is an emergent side effect of getting really good at "what word comes next" over a huge amount of text. The magic isn't a reasoning engine hiding inside; it's that predicting the next token well turns out to require a shocking amount of implicit knowledge.

Models don't see letters or words — they see tokens, which are sub-word chunks. "Strawberry" might be three tokens like straw + ber + ry. So when you ask "how many r's are in strawberry," the model never had the individual letters as separate units to count; it's reasoning over chunks that don't line up with characters. Same reason it fumbles reversing a string or doing tight character math. It's not dumb, it's just that its atoms aren't your atoms.

An embedding is a list of numbers that places a piece of text at a point in a high-dimensional space, arranged so things with similar meaning land near each other. "Dog" and "puppy" sit close; "dog" and "spreadsheet" sit far. Inside an LLM, every token gets embedded first — that's how text becomes something math can operate on. And it's the same trick that powers retrieval: to find relevant documents, you embed the query and the docs and look for the nearest points. I go deeper on that in the what is RAG piece.

When the model processes a word, attention lets it look back at every other word in the input and decide which ones matter for this one. In "the trophy didn't fit in the suitcase because it was too big," attention is what lets the model connect "it" to "trophy" rather than "suitcase." Each token essentially asks "who here is relevant to me?" and pulls in a weighted blend of the others. Stack that many times and the model builds a rich, context-aware sense of every word. That mechanism is the whole reason transformers beat what came before.

Training is the expensive, one-time-ish process where the model reads a mountain of text and adjusts its billions of weights so its predictions get better — that's where the knowledge gets baked in. Inference is what happens every time you actually use the model: weights are frozen, you feed in a prompt, and it predicts tokens out. Training costs millions and happens rarely; inference is the per-request cost you pay forever. As an engineer you live almost entirely in inference-land.

No — and this trips people up. The weights don't change when you chat. What looks like "remembering" earlier in a conversation is just that the whole conversation so far gets re-fed into the model as context on every turn. It's in-context, not in-weights. The moment that text falls out of the context window, it's gone. This is exactly why people confuse "give it examples in the prompt" with "fine-tuning" — one is temporary context, the other permanently changes the weights.

Prompting first, always — it's free and instant, so exhaust it before anything else. Reach for RAG when the problem is knowledge: facts the model doesn't have or that change often, like your company's docs. You retrieve the right text at question time and hand it over as context. Reach for fine-tuning when the problem is behaviour: a consistent format, tone, or a narrow task where you can't fit enough examples in the prompt. The classic mistake is fine-tuning to add facts — that's fragile and expensive; RAG is the tool for facts. I break the retrieval side down in the RAG interview questions post.

Remember the model outputs a probability distribution over next tokens. Temperature reshapes that distribution before it samples. Low temperature (near 0) makes it sharp — the model almost always grabs the single most likely token, so output is focused and repeatable. High temperature flattens it, so less likely tokens get a real shot, giving you variety and surprise. I use low temperature for extraction, classification, or anything where I want the same answer twice, and higher for brainstorming or copy. It does not make the model "more creative" in a smart way — it just widens the dice.

Both control randomness but from different angles. Temperature rescales the whole distribution. Top-p instead says "only consider the smallest set of tokens whose probabilities add up to p, and sample from those" — so it dynamically cuts off the long tail of unlikely garbage. With top-p at 0.9 you keep the plausible options and drop the nonsense, no matter how many tokens that turns out to be. In practice I usually tune one, not both, and top-p is a nice guardrail against the model wandering into weird low-probability tokens.

Because the model was trained to always produce a plausible next token, not to know when to stop. It has no built-in sense of "I don't actually know this" — so when it lacks the fact, it generates something that sounds right, because sounding right is literally what it was optimised for. There's no lookup step where it checks a source unless you add one. So hallucination isn't a bug you patch out; it's the flip side of the model's whole design. You reduce it — ground it with retrieval, tell it to say "I don't know," show citations, verify — but honest engineers say "reduce and detect," not "eliminate."

The context window is the maximum amount of text — prompt plus the model's own output — the model can consider at once, measured in tokens. Everything the model "knows" in that moment has to fit inside it. Context rot is what happens as you cram it full: models get noticeably worse at using information buried in the middle of a long context, and stuffing in more marginally-relevant text can actively hurt answers rather than help. Bigger windows are great, but "just paste everything in" is a trap. Curating fewer, more relevant tokens usually beats dumping in more.

Two reasons. First, sampling: if temperature or top-p allow randomness, the model rolls the dice on which token to pick, so runs differ by design — set temperature to 0 for near-determinism. Second, even at temperature 0 you can see tiny variation from floating-point and hardware nondeterminism, or because the provider quietly updated the model behind the same name. If your app needs reproducibility, pin what you can, log inputs and outputs, and don't assume "same prompt" guarantees "same output."

A stack of levers, cheapest first. Use a smaller, faster model for easy requests and escalate to a big one only when needed — routing beats one-size-fits-all. Trim the prompt: shorter context is faster and cheaper, and it dodges context rot. Cache aggressively — identical or near-identical requests shouldn't hit the model twice, and providers offer prompt caching for the static parts. Stream tokens so the user sees output immediately even if the full answer takes a beat. And always measure cost-per-request; a feature that's brilliant but negative-margin doesn't ship.

"It looked fine in the demo" is where things go to die. Build a small evaluation set — real inputs paired with what a good output looks like — and score against it every time you change the prompt or model, so you can see a number move instead of trusting vibes. For open-ended tasks, an LLM-as-judge with a clear rubric works surprisingly well. The point is the same as tests in normal code: without evals, every change is a guess and every regression is a surprise your users find first.

An agent is an LLM in a loop that can call tools. Instead of just answering, it decides "to do this, I need to search the web / run this code / hit this API," calls the tool, reads the result, and keeps going until it's done. The model provides the reasoning and the tool use provides the hands. The hard parts aren't the idea — they're keeping it from looping forever, handling tool failures gracefully, and stopping it from confidently doing the wrong thing. Great for multi-step tasks; overkill for a one-shot answer.