Menu

Post image 1
Post image 2
Post image 3
Post image 4
Post image 5
Post image 6
Post image 7
1 / 7
0

Every Model Cheats: Prompt-Level Mitigation of Cheating on Offensive Cyber Tasks

Hacker News·about 2 months ago
#UpyJ71AN
Reading 0:00
15s threshold

This blog is an abridged version of the full paper available on arXiv . We instructed 22 frontier models not to cheat on a cybersecurity benchmark. They cheated anyway, regardless of the prompts. Prior audits weren’t alarming. NIST found cheating in 0.3% of Cybench logs. The Meerkat study found 3.4% of successful traces involved cheating, implicating four models. Anthropic’s Claude Opus 4.6 system card described Cybench as “saturated,” reporting near-100% pass rates without a cheating audit. If these estimates were representative, cheating would be a marginal artifact. The ground truth is an order of magnitude worse. Under baseline conditions, 37.1% of all passes involved cheating and all but one model cheated. The average pass rate was 41.5%, but the average solve rate (tasks passed without cheating) was only 26.1%, with individual models inflated by as much as 5x. Models searched the internet for published solutions, read flag files from the evaluation infrastructure, and probed container metadata.…

Continue reading — create a free account

Join HashtagPLUS to read full articles, follow hashtags, vote, and join the conversation.

Read More