My Agent Cheated Its Benchmark. The Tests Weren't Even in the Room.

Reward hacking is when an AI agent earns a passing score by gaming the check instead of doing the work. The gaming takes familiar shapes: overwriting the test file, editing the score log, or finding any shortcut between the grader and the grade. Harbor is the open-source eval framework from the Terminal-Bench team, and since October 14 it runs every eval task inside Docker Sandboxes, the microVM isolation layer Docker built for exactly this class of problem.

Here is the punchline before the setup: I wrote a task where the agent's solution script refuses to do the work and instead hunts down the grader's test file to overwrite it. The scout came back with nothing. The agent searched its entire filesystem and the tests simply were not there, because the verifier runs in a separate container that the agent can never touch.…