Results Analysis Effort Setup 1

GPT-6 Astra

Codex CLI

Resolution rate: 46.25%

2

Fable 5.1

Claude Code

Resolution rate: 45.00%

3

Gemini 3.8 Flash

Gemini CLI

Resolution rate: 38.75%

4

GLM 5.3

Claude Code

Resolution rate: 37.50%

5

Muse Spark 1.3

Muse Code

Resolution rate: 36.25%

6

Grok 4.6

Grok Build

Resolution rate: 32.50%

7

GPT-5.6 Sol

Codex CLI

Resolution rate: 26.25%

8

Kimi K3

Kimi Code

Resolution rate: 20.00%

# Model Harness Resolution rate

1 GPT-6 Astra Codex CLI 46.25%

2 Fable 5.1 Claude Code 45.00%

3 Gemini 3.8 Flash Gemini CLI 38.75%

4 GLM 5.3 Claude Code 37.50%

5 Muse Spark 1.3 Muse Code 36.25%

6 Grok 4.6 Grok Build 32.50%

7 GPT-5.6 Sol Codex CLI 26.25%

8 Kimi K3 Kimi Code 20.00%

Resolution rate is equivalent to pass@1, averaged over eight independent runs per task. 95% confidence intervals are shown.

01 Introduction

Today we are releasing Real-SWE, a benchmark that evaluates frontier AI models on private, real-world, enterprise codebases.…