Part 1: The Problem
The following scenario is exaggerated, but typical.
A mid-sized software company introduces an AI assistant in its helpdesk. The setup is quick to build: a ticket comes in, a prompt is assembled with context from the knowledge base, the LLM writes a draft reply, and the agent reviews and sends it. The first weeks are a success. New hires work faster and the queue shrinks. This matches the research: in a large field study with more than 5,000 support agents, workers with AI assistance resolved 14% more issues per hour on average, and novices 34% more.
A few months later the mood shifts. The assistant keeps making the same mistakes:
Bank chargebacks end up in the regular refund queue.
The tone with angry customers is off, and every reply gets rewritten.
After a pricing change, it quotes the old terms for weeks.
Agents correct the same things every day, but the system never notices.…