Menu

Your agent passed 10,000 evals. It can still deadlock.
📰

Your agent passed 10,000 evals. It can still deadlock.

DEV Community: rust·IdeaSniper·about 6 hours ago
#E6tGxuBd
#devtorust#devto#dev#try#graph
Reading 0:00
15s threshold

If your agent workflow has 20 routing variables — retry counters, confidence thresholds, tool-success flags, budget checks — the guards on its edges combine in 2^20 = 1,048,576 ways. Run 10,000 evals and you have sampled under 1% of them (0.95%, to be exact). The other 99% is where the stalls live.

A green eval suite proves the workflow can work. It says nothing about the combinations you didn't sample — including the ones where every outgoing edge of some node is closed at the same time and the agent has nowhere to go.

The failure mode

This is an illustrative example, but the shape is common. Take a research agent with a critic loop: a planner calls a tool executor, a critic node checks the result, and two edges leave the critic. One loops back for another attempt while retry_count < 3 . The other exits — and whoever wrote it copy-pasted the same guard, so the exit also requires retry_count < 3 .

Every node is correct in isolation.…

Continue reading — create a free account

Join HashtagPLUS to read full articles, follow hashtags, vote, and join the conversation.