Real-SWE benchmark exposes weak AI coding agents

The best result in the Real-SWE benchmark resolved 38.8% of tasks drawn from private commercial codebases. Fable 5.1 reached the top score through Claude Code, followed by GPT-6 Astra through Codex CLI at 33.8%, Grok 4.6 through Grok Build at 32.5%, and Gemini 3.8 Flash through Gemini CLI at 31.2%.

Specific Labs designed Real-SWE around work from companies with active products and demanding production systems. The source code includes a consumer application serving more than 200,000 users, a fintech platform processing more than 100,000 bank statements, and enterprise sales systems. Reference solutions changed a median of 11 files, compared with six files in FrontierCode and DeepSWE.

The benchmark measures complete model-and-harness pairings. Each pairing used its native coding tool and high reasoning settings. Resolution rate equals pass@1 averaged over eight independent runs per task, with 95% confidence intervals.

Results varied sharply by task. API key and environment work reached a 71.9% aggregate resolution rate, while six of ten analyzed tasks finished below 15%. Billing schedule migration scored 14.1%, API token metering scored 12.5%, and tax jurisdiction logic scored 3.1%. An analytics stream reducer produced no successful solution across 64 attempts.

Rollout duration offered little separation. Agents failed 71.4% of runs lasting less than ten minutes and 73.4% of longer runs. Specific Labs traced many failures to missed requirements, integration errors, and assumptions left unverified inside the workspace.

Across failed runs, Fable 5.1 missed requirements in 36.7% and produced integration errors in 34.7%. GPT-6 Astra recorded unverified assumptions and integration errors in 34% of failed runs for each category.