A bug report lands: one file with a very long expression crashes the whole scan, with no file name and no summary. You add a guard, write a test with a file that crashes, watch it pass, close the ticket.
How do you find out that the guard has a hole? Not from the test. The test is the exact file from the report, and the guard was shaped around it.
I ran into this with a CLI that reads a codebase and lists the analytics events it can send, and the small SDK next to it that sends them. Both are on npm, both had a few hundred tests, and I was fairly sure they were in decent shape. So I asked for a different kind of check: three AI reviewers who had never seen my code or my reasoning. Each started from the README and the published packages, installed them into empty folders the way a stranger would, and tried to make them misbehave. I told them to distrust my tests and my changelog, and to run every finding at least twice before reporting it.…