Menu

#ContamiNation

7 posts

Feed·
7 of 7 posts
Why SWE-bench Verified no longer measures frontier coding capabilities
📰
0Read More

Why SWE-bench Verified no longer measures frontier coding capabilities

OpenAI·@HashtagPLUS·6 months ago
#XZvFSBv7
#background#too#contamination#bench#verified

SWE-bench Verified is increasingly contaminated and mismeasures frontier coding progress. Our analysis shows flawed tests and training leakage. We recommend SWE-bench Pro.

15s