Since we first published SWE-bench Verified in August 2024, the industry has widely used it to measure the progress of models on autonomous software engineering tasks. After its release, SWE-bench Verified provided a strong signal of capability progress and became a standard metric reported in frontier model releases. Tracking and forecasting progress of these capabilities is also an important part of OpenAI’s Preparedness Framework . When we created the Verified benchmark initially, we attempted to solve issues in the original evaluation that made certain tasks impossible to accomplish in the SWE-bench dataset ⁠ (opens in a new window) .

After initial leaps, state-of-the-art progress on SWE-bench Verified has slowed, improving ⁠ (opens in a new window) from 74.9% to 80.9% in the last 6 months.…