If you have put an LLM agent anywhere near customers, you already know the uncomfortable part. You can see everything it does, and you still don’t know whether it’s right.
Our team had full tracing on our support triage agent from day one. We could replay any ticket, read every tool call, watch it reason. What we didn’t have, for the first six weeks, was any way to say before a release whether the new version was better or worse than the old one. We found out in production, when return requests started landing in the wrong queue.
This post is the practical version of how we fixed that: how a 14-engineer team with no ML specialists built an eval set that now gates every change to the agent. Nothing here needs a platform. It needs a spreadsheet, a few hours from the person who knows the domain, and the discipline to run it every time.
Start with real tickets, not invented ones
The first instinct is to write test cases. Don’t.…