Agent loops that fix failing tests usually stop on one signal: the test command exits with 0. It's a good signal because a model can't argue with it. The osFoundry desktop app's starter loop "Get the JS suite green" verifies each run with npm test , and a comment in its Rust source ( loop_config.rs ) calls a command check "the only form that cannot be talked into passing".

That holds for the model's words. It doesn't hold for the files the model can edit. The agent fixing your tests has write access to the repository, and the exit code of npm test is produced by that repository: the test files, package.json , and the command line you wrote. Below, each way an exit code can say "pass" while the bug is still there is built as a tiny Node project and run.

Thirteen cases, measured

Each case is a fresh temp folder with src/add.mjs and a node --test suite. "Bug there" means add returns a - b and the test expects add(2, 3) to be 5. The script runs the command through sh -c and records the exit code.…