LLM System Design Interview: How to Prepare (2026)

The LLM system design interview looks like a normal system design round until about minute five, when you realise the load balancer isn't the hard part. You're designing around a probabilistic component that can be wrong, slow, and expensive all at once — and the interviewer wants to see whether you've internalised that or whether you're going to sketch a database and call it done.

I've sat on both sides of this. The candidates who do well aren't the ones who name-drop the most vector databases. They're the ones who treat retrieval, evaluation, and cost as first-class parts of the design instead of afterthoughts. Below is how the round actually differs, a framework you can reuse under pressure, a worked example, and the stuff people forget until the interviewer drags it out of them.

How this differs from a classic system-design interview

In a classic round, the model of computation is deterministic. You reason about queries per second, sharding, replication, consistency, caches. Given the same input, the system returns the same output, and correctness is basically a given — you're optimising for scale and availability.

An LLM system design interview keeps all of that and then hands you a component that's probabilistic. Same prompt, different day, different answer. It can hallucinate a confident wrong answer. Its "latency" is measured in tokens generated, not just a round-trip. And every call costs real money that scales with input and output length. So the design questions shift:

• Correctness isn't free. You have to design how you'll know the system is right — evaluation and feedback loops — because you can't assume it.
• Knowledge is a design decision. Where do facts come from? The model's frozen weights, retrieval at query time, or a tool call? That choice drives your whole architecture.
• Latency is token-shaped. A long answer streams out over seconds. You design for time-to-first-token and streaming, not just p99 of a single response.
• Cost is per-query and visible. Nobody used to ask the cost of one search query in an interview. Here, "what does one request cost?" is a fair and common question.
• Safety is in scope. Prompt injection, leaking context, toxic output — these are architectural concerns, not a checkbox at the end.

If you walk in treating the LLM as a black box that "just answers," you'll design a system that works in the demo and falls over in production. Interviewers know this, and it's exactly what they're probing.

A framework you can reuse

Under pressure, having a script beats improvising. Here's the seven-step spine I'd walk through out loud for almost any LLM design question. Say the steps, then go deep where the interviewer leans in.

• 1. Clarify requirements and scope. Who's the user? What's the task — Q&A, summarisation, an agent that takes actions? What's the volume, the latency budget, the accuracy bar, the budget per query? Is the knowledge public or private? Pin these down before you draw a single box; a huge amount of the design falls out of the answers.
• 2. Data and retrieval. Where do the facts live, and how do they get to the model? If it's grounded in private docs, you're building retrieval: ingestion, chunking, embeddings, a vector store, top-k search, maybe a re-ranker. Decide what's indexed offline versus fetched live.
• 3. Model choice and prompt strategy. One big model or a cheap-model-with-escalation setup? Hosted API or self-hosted? What goes in the system prompt, how do you structure context, do you need structured output or tool-calling? Justify the tradeoff — you're rarely marked down for a choice you can defend.
• 4. Serving: latency, streaming, caching. How does a request flow through the system? Stream tokens so time-to-first-token feels instant. Cache embeddings, frequent queries, and (carefully) responses. Handle timeouts and retries around a slow model call.
• 5. Evaluation and feedback loop. How do you know it's working, and how does it get better? Offline evals on a golden set, online metrics (thumbs, deflection, escalation rate), and a loop that turns real failures into new test cases. This is the step that separates senior from mid.
• 6. Cost and monitoring. Cost per query and total spend, token usage, latency percentiles, retrieval hit rate, error and fallback rates. Design the dashboard you'd actually stare at during an incident.
• 7. Safety, guardrails, and failure modes. Input filtering and prompt-injection defence, output moderation, PII handling, and — crucially — what happens when the model is down, times out, or has nothing to say. A graceful "I don't know" is a design feature.

You won't spend equal time on all seven, and you shouldn't. But naming them signals that you know the whole surface, and then you can dive where it matters.

A worked example: design a RAG chatbot over a company's docs

Let's walk the framework on the most common prompt you'll get: design a chatbot that answers employee questions from an internal documentation set. If retrieval feels shaky, my RAG interview questions guide drills the retrieval half harder than I can here.

Clarify first. I'd ask: how many docs, and how often do they change? How many users and queries per day? What's the latency budget — is a two-second answer fine? What's the accuracy bar, and what's the cost of a wrong answer (an HR policy mistake is worse than a cafeteria-menu mistake)? Should it say "I don't know" rather than guess? Say the docs are tens of thousands of pages, updated daily, a few thousand queries a day, and wrong answers are costly. That shapes everything.

Data and retrieval. Offline ingestion: pull docs, chunk them along natural structure (headings, sections) at a few hundred tokens with slight overlap, embed each chunk, store vectors plus source text in a vector database. Online: embed the question, retrieve top-k, run a re-ranker so the best chunk isn't buried at position eight, and pass the survivors to the model. I'd go hybrid — dense vectors plus keyword search — because internal docs are full of exact terms (error codes, system names) that dense retrieval alone whiffs on.

Model and prompt. A capable hosted model, prompted to answer only from the retrieved context and to say it doesn't have the information when the context doesn't cover the question. Include citations to source chunks so answers are verifiable. For volume I'd consider routing easy questions to a cheaper model and escalating hard ones.

Serving. Stream the answer so it feels instant. Cache question embeddings and cache answers for common, stable questions (with a short TTL so a doc update doesn't serve stale policy). Wrap the model call in a timeout with a graceful fallback message.

Evaluation. Build a golden set of question → correct-source pairs. Measure retrieval hit rate (is the right chunk even showing up?) and generation faithfulness (did the answer stick to the sources or invent?). Wire up thumbs-up/down in the UI and funnel every thumbs-down into the golden set. Now I can change chunking or swap a model and watch a number move instead of arguing about vibes.

Cost and monitoring. Track cost per query, token usage, p50/p95 latency, retrieval hit rate, and the rate of "I don't know" responses (a spike means retrieval or ingestion broke). Alert on cost and latency, not just errors.

Safety and failure modes. Strip and filter user input to blunt prompt injection, never let retrieved content override the system instruction blindly, and moderate output. If retrieval returns nothing relevant, the bot says so instead of hallucinating. If the model is down, a friendly fallback and a logged incident — never a fabricated policy.

Notice I never had to invent a fictional benchmark or company. Every claim is a design decision you can defend, which is exactly the tone that lands.

The things candidates forget

These are the gaps I watch people fall into, in rough order of how often it happens:

• Evaluation. The single biggest miss. People design a beautiful pipeline and never say how they'd know it works. If you bring up a golden set and faithfulness metrics unprompted, you've already separated yourself from most candidates.
• Cost per query. "How much does one request cost?" catches people flat. Have a rough mental model: input tokens plus output tokens times a price, multiplied by traffic. Know the levers — fewer/better chunks, smaller models for easy queries, caching.
• Freshness and ingestion. Everyone designs the read path; few design the write path. How do docs get in, get updated, get deleted? Stale facts lingering because you never handled deletes is a real, embarrassing bug.
• Guardrails. Prompt injection and data leakage aren't hypothetical when your context contains private docs. Mention input filtering and not trusting retrieved text as instructions.
• The graceful "I don't know." A system that confidently makes something up is worse than one that admits a gap. Designing the abstain path — and monitoring how often it fires — is a senior tell.

You don't need to cover all five in depth. But leaving all of them out is how a fine-sounding design quietly fails the round.

How to practice

Reading a framework isn't the same as producing one at a whiteboard while someone interrupts you. A few things that actually move the needle:

• Talk out loud, on a timer. Pick a prompt — "design a support bot," "design a code-review assistant," "design a meeting summariser" — and walk the seven steps aloud in 30 minutes. The friction of speaking it exposes the parts you only think you understand.
• Actually build one. Nothing makes retrieval, chunking, and evals concrete like watching your own RAG bot return garbage and having to fix it. The debugging instinct you build translates straight into interview answers.
• Get the fundamentals solid first. System design sits on top of the building blocks — embeddings, tokens, context windows, tool-calling. My AI engineer interview questions post covers the layer underneath this one.
• Rehearse the tradeoffs, not the diagram. Interviewers reward "I'd pick X over Y because Z." Practise defending choices, not memorising one canonical architecture.

Here's a compact prompt set you can run against yourself or a friend:

Open-source companion: Awesome AI Engineer Interview Questions — 105 curated questions on GitHub, free.