Menu

Real-SWE Benchmark — Specific Labs
📰
0

Real-SWE Benchmark — Specific Labs

withspecific.com·Snagnik Das·24 days ago
#Mm5o6IoO
Reading 0:00
15s threshold

Results Analysis Effort Setup 1 GPT-6 Astra Codex CLI Resolution rate: 46.25% 2 Fable 5.1 Claude Code Resolution rate: 45.00% 3 Gemini 3.8 Flash Gemini CLI Resolution rate: 38.75% 4 GLM 5.3 Claude Code Resolution rate: 37.50% 5 Muse Spark 1.3 Muse Code Resolution rate: 36.25% 6 Grok 4.6 Grok Build Resolution rate: 32.50% 7 GPT-5.6 Sol Codex CLI Resolution rate: 26.25% 8 Kimi K3 Kimi Code Resolution rate: 20.00% # Model Harness Resolution rate 1 GPT-6 Astra Codex CLI 46.25% 2 Fable 5.1 Claude Code 45.00% 3 Gemini 3.8 Flash Gemini CLI 38.75% 4 GLM 5.3 Claude Code 37.50% 5 Muse Spark 1.3 Muse Code 36.25% 6 Grok 4.6 Grok Build 32.50% 7 GPT-5.6 Sol Codex CLI 26.25% 8 Kimi K3 Kimi Code 20.00% Resolution rate is equivalent to pass@1, averaged over eight independent runs per task. 95% confidence intervals are shown. 01 Introduction Today we are releasing Real-SWE, a benchmark that evaluates frontier AI models on private, real-world, enterprise codebases.…

Continue reading — create a free account

Join HashtagPLUS to read full articles, follow hashtags, vote, and join the conversation.

Read More