A founder asks an AI agent to recommend which customer accounts deserve more attention. The agent checks revenue, support costs and renewal dates, then returns a sensible ranking. The numbers are accurate. The founder reviews the list and approves it.
But somewhere along the way, the agent decided what “deserve more attention” meant. Perhaps it prioritised immediate revenue over customers with growth potential. Perhaps it treated support requests as costs rather than evidence that a customer needed help.
The founder checked the answer. The objective behind it received less attention.
This is a harder problem than catching a hallucination. Nothing needs to be false for a recommendation to carry a choice that the person receiving it never consciously made.
When the calculation works
Anthropic published research on 27 October examining how agents influence experts’ judgement calls. Fifty professional analysts used Claude Code to assess a farming advisory programme.…