Many quality engineering teams have started using agentic AI testing. They connect an agent to their app and give it a human-readable task like “Test the checkout flow and tell me if anything is broken.” And it often works great! A few minutes later, they have their answer.
However, if they send that same request tomorrow, even with the same exact code set, that “great” answer might just change.
This happens because the agent takes an alternate route through checkout. Or tests different details. Or uses different test data. Regardless of the why , the test just isn’t consistent. In fact, in a recent experiment, Slack’s engineering team showed that only about 20% of runs follow the exact same sequence of actions … even with the same detailed prompts!
This is a problem for regression tests and audits, or anything else that needs to be consistent.…