Menu

1 / 9

A 100% AI Eval Score Is a Red Flag, Not a Win

DEV Community: machinelearning·Max Quimby·14 days ago
#EyWSX98U
Reading 0:00
15s threshold

A 100% AI Eval Score Is a Red Flag, Not a Win

📖 Read the full version with charts and embedded sources on ComputeLeap →

In April 2026, a team at UC Berkeley's RDI lab ran an automated scanning agent against eight of the most-cited AI benchmarks in the industry — SWE-bench Verified, SWE-bench Pro, Terminal-Bench, WebArena, FieldWorkArena, CAR-bench, GAIA, and OSWorld. The agent scored 100% on six of them . It didn't solve a single task. On SWE-bench Verified — the benchmark OpenAI, Anthropic, and Google have used to compare their coding agents — the exploit was a pytest conftest.py hook that marks every test as "passed." The entire attack surface: the agent and the evaluator ran in the same container.

This is not a theoretical vulnerability. This is the evaluation infrastructure the AI industry currently relies on to compare billion-dollar models.…

Continue reading — create a free account

Join HashtagPLUS to read full articles, follow hashtags, vote, and join the conversation.