Three of our checks gave the wrong answer on the same day, and one rule fixes all three: a check has to say what it looked at, and what it looked at has to be the real thing. A check on how much input each helper script reads answered PASS while the nine Perl helper scripts it was supposed to check went unread. A screenshot scanner reported green while its plugin did nothing. One test environment rejected a correct plugin, because the machine it ran on was not set up like a normal install.
If you trust a passing check to have looked at the thing it claims to check, the three failures below are worth your time. Each one is a different kind of check: one that reads files, one that reads a folder, and one that reads the environment it runs in.
How does a CI check pass without reading what it claims to check?
A CI (continuous integration) check can pass without reading what it claims to check because nothing required the verdict to name the files it opened.…