I keep seeing AI systems fail at state updates. They cache an answer, receive data that breaks it, and still fail to update. I wanted to find out why.

Say a travel bot books a flight, hotel, and cab. It marks the trip confirmed. Later, the flight gets canceled. The bot logs the cancellation. But does it cancel the cab? Does it update the hotel? Does it know your dentist appointment is unrelated and should stay put?

Updating beliefs is hard. A system needs to change some facts, keep others, and drop old assumptions. Getting one answer right on a standard benchmark does not prove a system can actually do this.

I built RevisionTrace to test it. It is a Python benchmark for belief revision.

It runs a test using two different routes. Route one gives the system all the final data upfront to calculate everything from scratch. Route two starts with a baseline state and feeds in live updates, forcing the system to adapt on the fly.…