A common interview mistake is to evaluate an AI agent by asking whether it produced a good answer once. That tests a demo, not a system.

A stronger answer explains how to define success, build an evaluation set, compare outputs against evidence, and monitor behavior in production. The interviewer is usually testing whether you can turn an open ended AI problem into a measurable engineering process.

Before naming metrics, clarify the agent's job. An agent that retrieves documents, calls tools, and completes a workflow has more evaluation dimensions than a single turn chatbot.

The interviewer is likely looking for four things:

Can you define task success from the user's perspective?

Can you separate intermediate behavior from the final answer?

Can you design reliable test cases and review procedures?

Can you detect regressions when prompts, models, tools, or retrieval settings change?

You do not need to list every possible metric.…