Last week, OpenAI evaluated two models on an exploit benchmark within an isolated sandbox. Guardrails were reduced for testing, and the models found a vulnerability in their environment, accessed the internet, and reached Hugging Face's production database.

No human directed the action, but the breach is a clear example of how much more capable malicious attackers are when equipped with powerful AI models. But defenders have the same tools, and a clear advantage: knowledge of their own codebase. Hacks are initiated from the outside, so the single best defense is finding vulnerabilities from the inside before attackers do.

Today we're releasing DeepsecBench , a benchmark that evaluates how well different models find cybersecurity vulnerabilities in application code. For each model the report includes recall, precision, cost, and total time, and combines recall and precision into a single benchmark score.…