An AI code reviewer should do more than identify suspicious code. It should also report validation results accurately.

For this test, I added the following JUnit assertion to a GitHub pull request:

@test
void shouldDetectIncorrectExpectedValue() {
assertEquals(5, 2 + 2);
}

The test is intentionally incorrect.

What CodeGuard AI reported:

Test execution: FAILED

Tests executed: 1

Tests failed: 1

Expected: 5

Actual: 4

Suggested fix: change the expected value to 4

It also identified the failing assertion in the changed file.

One finding isn't enough to establish overall reliability, so I'm testing CodeGuard against 100 scenarios covering failure detection, test-result evidence, false positives, merge policies, and more.

The next question is just as important: should a failing test automatically block a merge?

I'm building CodeGuard AI with Java 21 and Spring Boot. The source code is available on Gumroad.

What behavior would you expect from an AI reviewer when a test suite fails?