The Signal: The Legally Binding Hallucination
Recently, a major airline's customer support chatbot hallucinated a bereavement fare policy. A customer claimed the refund, the airline refused, and a tribunal ruled in favor of the customer. The chatbot was deemed a legal agent of the company.
The failure wasn't that the LLM hallucinated—it’s that it was allowed to speak directly to the customer and the database without a chaperone. When you give a non-deterministic guest unregulated access to your deterministic house, you are legally and financially responsible for the fire.
We need to stop treating AI as an open-ended "chat" interface and start treating it as untrusted, highly volatile code execution.
Phase 1: The Architectural Bet
We are shifting from Open Dialogue to Hardened State-Machine Confinement.
The Vendor Trap is the "Chat Completion API." It encourages you to build open text boxes where users ask for anything, and the AI returns anything.…