Agent Evaluation interview questions

Observability is descriptive: logs and traces of every step, tool call, and token — it tells you what happened and is great for debugging. Evaluation is the judgment layer on top: a repeatable way to score outcomes against what should have happened (pass/fail, a rubric, a number). Adoption is lopsided — roughly 89% observability vs 52% evaluation — so most teams can watch their agents but never actually test them, which is why so many feel fine in a demo and disappoint in the wild.

Agents are multi-step, so a single "was the final answer good?" check is too coarse. Trajectory evaluation scores the path: did it choose the right tool, retrieve the relevant document, avoid needless steps, and recover when something failed? The point is that two runs can land on the same answer — one through sound reasoning, one by luck — and only trajectory-level evals can tell them apart. That distinction is exactly what you need before trusting an agent with real actions.

A trace shows the agent called three tools and returned an answer — it does not tell you whether that was the right answer, the right tools, or the cheapest path. It surfaces a failure only after it ships, when an angry user hits it. An eval gate runs the change against a golden set in CI and fails the build if scores drop below baseline, so the regression never reaches production. Logging is discovery-after-the-fact; the gate is prevention.