AI Engineer Interview Questions (2026): What They Actually Ask

It predicts the next token. That's it — given a sequence of text, it outputs a probability distribution over what comes next, samples one, appends it, and repeats. Everything that feels like reasoning is that loop running at scale over a model trained on a huge amount of text. I like to say it out loud plainly because it kills two bad instincts at once: it's not a database looking things up, and it's not a mind that "knows" things. It's a very good pattern-continuation engine, which is exactly why it hallucinates and why grounding it with real context matters.

A token is a chunk of text the model actually operates on — usually a word, part of a word, or a punctuation mark, not a whole word and not a single character. "Interviewing" might be two or three tokens. You care because you pay per token, your context window is measured in tokens, and latency scales with how many you generate. When someone asks me to cut cost, tokens are the first place I look: shorter prompts, tighter retrieved context, and not asking the model to echo back a giant blob it doesn't need to.

It's a list of numbers that places a piece of text at a point in space, arranged so that things with similar meaning land near each other. "Refund policy" and "how do I get my money back" end up close even though they share no words. That's the whole trick behind semantic search — you turn the question and your documents into points, then find the nearest ones. The key insight I'd want to land: embeddings capture meaning, not keywords, which is both their strength and the reason they whiff on exact terms like error codes or SKUs.

Training knowledge is baked into the weights — frozen at whenever training stopped, and you can't see or edit it. The context window is the working memory you hand the model at request time: the prompt, the conversation so far, any documents you retrieved. Two very different failure modes fall out of this. If the model's parametric knowledge is stale or wrong, you fix it with retrieval or fine-tuning. If it "forgot" something from earlier in a long chat, that's a context-window problem — the relevant text fell out of the window or got buried.

Temperature controls how much randomness there is when the model picks the next token. Low temperature makes it pick the most likely option almost every time — deterministic, repetitive, safe. High temperature lets it wander, which you want for brainstorming or creative copy. For anything where I need structure — extracting fields, classifying, calling a tool with exact arguments — I turn it down near zero so I get the same answer twice and don't fight random formatting. It's a small dial that quietly fixes a lot of "why is the output flaky" complaints.

RAG adds knowledge at question time — you fetch relevant text and hand it to the model, like an open-book exam. Fine-tuning changes the model's behaviour or style by continuing to train it on your examples, like drilling a habit in. So the rule of thumb I actually use: if the problem is facts, especially facts that change, reach for RAG — you just update the index, no retraining. If the problem is form — always answering in a certain tone, format, or domain style — that's fine-tuning. And plenty of real systems do both. The wrong answer is picking one because it's the thing you know how to build.

Use an agent when the task genuinely needs multiple steps that you can't script ahead of time — the model has to decide what to do next based on what it just found, maybe call a few tools, maybe loop. Research assistants, "book me a flight," multi-step debugging. Do not use an agent when a single prompt or a fixed pipeline would do, because agents are slower, pricier, harder to test, and they compound errors — one wrong step early and the whole chain drifts. My honest default is: start with the simplest thing (one call, then a fixed chain), and only add agentic autonomy when you can point at a task that actually requires the model to make decisions in the loop.

Tool calling is when you describe some functions to the model — name, what they do, what arguments they take — and instead of answering in prose, the model can respond with "call this function with these arguments." Your code runs the function, feeds the result back, and the model continues. That's how an LLM checks live data, does math it's bad at, or takes an action. Where it breaks: the model hallucinates an argument, calls the wrong tool, or calls one when it should've just answered. So I validate every argument before executing, keep tool descriptions crisp and few, and never let a tool do something destructive without a guard. The model proposes; my code disposes.

The system prompt sets the standing rules — who the model is, what it must and mustn't do, the output format — and it sits above the back-and-forth so it stays in force across the whole conversation. User messages are the actual turns. Practically, I put durable constraints (tone, safety rules, "answer only from the provided context") in the system prompt and keep per-turn stuff in the user message. And I'll mention the security angle unprompted: never trust user text to override system rules — prompt injection is real, so I keep retrieved or user-supplied content clearly fenced off from instructions.

First I build a golden set — a couple dozen to a few hundred real inputs with the outcome I want. Then I measure against it every time I change a prompt or model, so I see a number move instead of trusting vibes. For anything with a clear right answer I check that directly. For open-ended output, I use an LLM-as-judge: a second model scored against a rubric I wrote — is it faithful to the source, does it actually answer, is the tone right. It's not perfect, but it's consistent and it scales, and I spot-check the judge against my own labels so I trust it. The whole point is turning "seems fine" into a metric I can regression-test.

Faithfulness means the answer only claims things that are actually supported by the context you gave the model — no smuggled-in facts. It's the specific flavour of hallucination that matters most in a RAG or summarization feature. To catch it, I have a judge model (or a checker) go claim by claim and ask "is this supported by the retrieved text?" Anything unsupported gets flagged. In the product I pair that with citations, so a human — or the user — can trace a sentence back to its source. Faithfulness is separate from relevance: an answer can be perfectly grounded and still not answer the question, so I measure both.

You reduce and detect — you never fully eliminate, and I'd say that plainly rather than over-promise. Reduce: give the model good context so it isn't forced to improvise, and instruct it to say "I don't have that information" when the context doesn't cover the question. Detect: run a faithfulness check that flags unsupported claims, and show citations so it's easy to verify. Turning temperature down helps on factual tasks too. The framing an interviewer wants is engineering, not magic: I make it rare, I make it visible, and I give the user a way to catch the ones that slip through.

I measure cost-per-request and latency first, because I refuse to guess. Then the usual levers: use a smaller, cheaper model for the easy cases and only escalate hard ones to the big model; cut the tokens — trim the prompt, retrieve fewer but better chunks, don't make the model echo huge inputs; cache repeated queries and embeddings; and stream the response so it feels fast even when total time is the same. I'll also push back on the product itself — sometimes the fix is "we don't need the model for this step at all." A feature that's accurate but loses money per call doesn't ship, and I want the interviewer to hear that I think about the bill.

This is where people lose the loop by rambling, so I keep a tight shape: the problem and who it was for, the approach and one real decision I had to make (why RAG here, why an agent there), the thing that went wrong, and how I knew it was fixed — ideally with a number. The "what went wrong and how I measured the fix" part is the whole answer. Anyone can describe a happy path from a tutorial. Saying "our retrieval was pulling the wrong passages, I logged the top chunks, saw the right one wasn't there, fixed the chunking, and hit rate went from the 60s to the 90s" — that's the sentence that gets you hired, because it proves you've debugged this for real.

I don't just say no, and I don't just build it. I ask what outcome they're after, then map it to whether an LLM is even the right tool — sometimes the answer is a rule, a search box, or a form, and the model adds cost, latency, and a failure mode for nothing. If it is a fit, I name the risks out loud (hallucination, cost, the eval work it'll need) so we go in with eyes open. Interviewers love this one because it's really asking: are you an engineer who reaches for the model reflexively, or someone who protects the product from a shiny bad idea.

Honestly, I stopped trying to chase every release. What I do instead is build on the parts that don't churn — retrieval, evaluation, prompt design, tool calling, cost thinking — because those transfer across every provider and every new model. Then I keep a small golden eval set so when a new model drops I can swap it in and measure whether it's actually better for my task instead of trusting the launch hype. And I read the primary sources — model and API docs — over hot takes. The meta-answer I want an interviewer to hear: I don't need to know today's benchmark leader, I need durable skills plus a way to test any model quickly. If you want the fuller version of that, it's basically our AI engineer roadmap.