Menu

1 / 2

My AI agent got worse after a prompt change. Nothing crashed. Here's how I catch that in CI now.

DEV Community: typescript·Yadnesh Deshpande·7 days ago
#PtSQT2qt
#devtotypescript#devto#dev#lookup_order#tool
Reading 0:00
15s threshold

Normal software fails loudly. An exception, a 500, a red test. AI agents usually don't. After a prompt tweak or a model upgrade, an agent can start:

calling the same tool three times in a row

skipping a step that matters, like checking a policy before acting

taking more steps, and costing more, for the same task The final answer might even look fine. Nothing errors, so nobody notices until a user hits the bad path or the bill arrives.

I wanted a test that fails when that happens, so I built it into agentcheck , an open-source TypeScript library and CLI.

The idea: test the trace, not just the answer

An agent's behavior is the sequence of tool calls it makes. If you save that sequence for a run you trust, you can check new runs against rules and against that baseline.

Take a refund agent.…

Continue reading — create a free account

Join HashtagPLUS to read full articles, follow hashtags, vote, and join the conversation.