Tools that turn a website into an API for an AI agent often generate Playwright code and repair it when it fails. On a recent Launch HN thread , commenters asked two questions about this: can a tool return success when the action never completed, and does anything notice when a field goes empty while the response still looks fine?
I measured it. I changed an orders page in 10 ways and ran 3 AI-generated tools against the baseline and each changed version, with a repair loop. In 16 of 66 runs (16 of the 60 on a changed page) the tool returned wrong data and reported success. The repair loop never saw those runs, because it only starts on an error. A schema check plus four checks against the page turned all 16 into errors, and the repair loop could then work on them.
The code, every generated tool and every run are in the PoC repo .
The setup
The site is a small orders page: 12 orders, 8 of them open, with a Cancel button on each row. The page loads its data from an API.…