Menu

Mutate the Grader, Not the Code: A Weekly Self-Audit for AI C++ Patch Evals
πŸ“°

Mutate the Grader, Not the Code: A Weekly Self-Audit for AI C++ Patch Evals

DEV Community: cppΒ·Finley LiΒ·6 days ago
#uu2ZbyCy
#devtocpp#devto#dev#grader#audit
Reading 0:00
15s threshold

Example scenario (illustrative, not a reported incident): a team's eval harness has returned PASS on every AI-generated C++ patch for three weeks. Then a patch replaces a bounded comparison with a modulo on a signed value. The unit grader stays green because no golden case feeds a negative operand. The sanitizer grader stays green because it runs the same test binary, which never reaches the branch. Nothing in the report is lying β€” the harness simply stopped being able to see that class of bug.

Green means "the cases we wrote still pass". It does not mean "the grader can still detect this failure". Those are different properties, and only the second one decays silently.

The technique below audits the second property directly: seeded faulty patches, known as mutants, are pushed through the real harness, and every grader that fails to reject them is reported as blind. No model names, quotas, or benchmark numbers are assumed; the artifact is a plain script and a manifest.

1.…

Continue reading β€” create a free account

Join HashtagPLUS to read full articles, follow hashtags, vote, and join the conversation.