An AI agent can make a perfectly reasonable decision and still get things wrong. Imagine an agent managing a crypto wallet. It checks the balance, reviews a transaction request, verifies that the amount is within the permitted limit, and prepares a transfer. Everything looks fine. The agent has followed its instructions, the calculations are correct, and the transaction appears to satisfy the application's rules.
But there is a problem. Another transaction has already moved some of the funds, and the application's internal balance has not been updated yet. The agent is working with yesterday's reality, even though the data it received looks legitimate.
This is the kind of problem that interests me when thinking about production AI systems. We spend a lot of time discussing better models, retrieval quality, context windows, and tool calling.…