The Coding Benchmark That Finally Matched the Work

For a while, coding-agent leaderboards had a strange property: the more I used the models, the less I recognized the rankings.

Models separated by a few points on a benchmark could feel several generations apart inside a repository. One would keep the whole change coherent, follow the awkward requirement hiding in paragraph four, test its own work, and recover when the first approach failed. Another would produce a plausible patch that left me doing the difficult half of the job.

Two comparisons made the problem especially visible. Artificial Analysis’s now-superseded Terminal-Bench Hard evaluation reported Kimi K2.7 Code at 44.7% pass@1 and Claude Opus 4.6 max at 46.2%. Both models ran under the same Terminus 2 harness. In my repository work—not a controlled evaluation, but the work behind my buying decision—the practical gap was much larger. GPT-5.5 against Opus 4.7 was the sharper example: the reported SWE-Bench Pro Public comparison put Opus ahead, while my experiments strongly favored GPT-5.5. Later work with Opus 4.8 did not change that judgment.

DeepSWE supplied the missing explanation. Its first leaderboard put GPT-5.5 clearly in front. More importantly, it put the rest of the models in roughly the order I had already formed by using them. It was the first coding benchmark that felt less like an abstract score and more like a measurement of the work.

The benchmarks were answering a different question

This is not an argument that SWE-bench and its descendants are useless. They moved coding evaluation from toy functions to changes in real repositories. That was an enormous improvement. The problem starts when a useful research benchmark becomes a model-buying guide without anyone checking whether it measures the same thing as day-to-day agent work.

The later benchmarks also fixed real weaknesses. SWE-bench Verified added human screening. SWE-Bench Pro made the tasks longer, augmented their specifications, and introduced private and held-out splits. Those are meaningful advances, not paperwork. They just do not remove every consequence of mining historical fixes and inheriting tests written for a known patch.

My practical question is not “can this system sometimes produce a patch that satisfies the tests attached to an old GitHub issue?” It is closer to this: can I give the agent a meaningful change in an unfamiliar repository, let it explore, implement, validate, and repair, then get back something I would seriously consider merging?

Several properties of popular benchmarks weaken that connection.

1. Public answers can leak into training data

The original SWE-bench family is built from public issues and the commits that resolved them. The problem statement, discussion, implementation, and tests can all end up in training data. A passing run may demonstrate problem-solving, recall, retrieval, or some mixture of the three. The score does not tell us which.

2. Patch tests make imperfect general graders

Tests committed with a pull request were written to protect that implementation and that project. They were not designed to evaluate every valid solution a future agent might invent. They can reject a correct alternative because it does not introduce the same private helper, or accept an incomplete implementation because the original patch never needed to exercise the missing path.

The DeepSWE paper reports an independent-judge disagreement rate of 32.4% for the sampled SWE-Bench Pro rollouts, against 1.4% for DeepSWE. That is disagreement, not a proven ground-truth error rate, and the auditor was itself an LLM. Still, a one-in-three disagreement is large enough to make a leaderboard wobble.

3. Saturated benchmarks compress frontier models

Once many models solve most of a benchmark, small score differences carry an unreasonable amount of meaning. Confidence intervals overlap. Harness revisions, prompt details, and a handful of tasks can reorder the top of the table. The chart says “near peers” while the experience of supervising them says otherwise.

DeepSWE compared eight model configurations that spanned about 30 percentage points on publicly reported SWE-Bench Pro results. On DeepSWE, the same configurations spanned about 70 points. A wider range is not automatically a better benchmark—you can create separation with bad questions too—but it gives a benchmark room to represent real capability differences instead of squeezing them into statistical noise.

4. A model is not its agent product

Codex CLI, Claude Code, Cursor, and Gemini CLI have different prompts, editing tools, context management, and control loops. A leaderboard that mixes models and scaffolds may be ranking the combined systems. A leaderboard that fixes the scaffold is cleaner for comparing models, but no longer reproduces the products developers actually use. There is no free apples-to-apples lunch here.

Claude was the tell

The May 26, 2026 DeepSWE publication snapshot captured the mismatch almost perfectly. Its eight-model cross-benchmark comparison supplies the four selected rows below. The SWE-Bench Pro Public figures came from vendor announcements and Scale Labs’ public leaderboard; the DeepSWE figures came from runs under its fixed mini-swe-agent harness and shared prompt. The rows align models, not evaluation stacks, and the effort labels describe only the DeepSWE runs. This is a comparison of published orderings, not a controlled rerun.

In the reported SWE-Bench Pro Public results, Opus 4.7 led GPT-5.5 and Sonnet 4.6 looked close behind. DeepSWE put GPT-5.5 sixteen points ahead of Opus and thirty-eight ahead of Sonnet. That second ordering looked far more like my own experience.

This was not universal across every existing benchmark. Terminal-Bench 2.0 had already put GPT-5.5 ahead of Opus 4.7. My claim is narrower: DeepSWE was the first benchmark whose full ordering looked natural to me, and whose task shape explained why.

My experiments were not a publishable benchmark. They were biased toward the repositories, languages, prompts, and standards I care about. But that is exactly why the mismatch mattered: I was trying to choose an agent for my work. A benchmark can be scientifically useful and still be a poor predictor for that decision.

DeepSWE did not prove that my judgment was right. It supplied a better explanation for why the older rankings felt wrong.

Why DeepSWE tracks the work better

DeepSWE changes more than the difficulty knob. Its 113 tasks are original changes written for the benchmark across 91 active repositories and five languages. The reference solutions are not adapted from existing commits and are not merged upstream. Before the benchmark was released, there was no public gold patch for a model to remember or discover.

The tasks are also shaped more like sustained engineering work. The published corpus statistics report an average reference solution of 668 added lines across seven files. SWE-Bench Pro averages 120 lines across five files; SWE-bench Verified, ten lines in one file. Solution size is not the same as difficulty, but larger multi-file changes ask much more of an agent’s ability to explore a repository, retain a specification, coordinate edits, and verify the whole result.

Each DeepSWE task gets a purpose-written functional verifier. It checks observable behavior instead of requiring the agent to reconstruct the reference patch’s internal shape. In v1.1, the benchmark also extracts the agent’s committed diff and grades it in a clean verifier container, which closes several ways an agent could accidentally or deliberately influence its own test environment.

Finally, every model runs through the same mini-swe-agent harness with one bash tool and a shared prompt. That sacrifices some product realism, but it removes a large confound when the question is model capability. DeepSWE is not comparing Codex CLI with Claude Code; it is asking what the underlying models do when given the same loop.

These choices plausibly explain the alignment I saw. Original tasks reward novel problem-solving. Larger changes punish agents that lose requirements halfway through. Behavioral verifiers reward a correct alternative rather than fidelity to a hidden patch. The benchmark is measuring more of the failure modes I pay for in practice.

First GPT-5.5, now GPT-5.6

GPT-5.5 was the striking result in the original release: 70% pass@1, fourteen points above GPT-5.4 and sixteen above Claude Opus 4.7. On a benchmark whose task design produced a much wider score range, it sat clearly apart.

The DeepSWE v1.1 leaderboard was published June 14 and updated July 17. It now has GPT-5.6 Sol at 73%, Claude Fable 5 and GPT-5.6 Terra at 70%, Kimi K3 at 69%, and GPT-5.6 Luna and GPT-5.5 at 67% when comparing the highest-effort configurations. It also puts Opus 4.8 at 59%. That last comparison is much closer to what I saw when I put GPT-5.5 against Opus 4.8 myself.

The confidence intervals of the top configurations overlap, so 73 versus 70 is not evidence of a clean statistical victory. It is fair to say Sol leads the displayed point estimate; it is not fair to pretend the benchmark can reliably order every model separated by three points.

There is an important chronological asterisk. GPT-5.5 shipped before DeepSWE published its tasks, verifiers, and reference solutions, making that first result the cleaner test of novel problem-solving. GPT-5.6 shipped after those artifacts were public. Without a disclosed training cutoff, I cannot assume its DeepSWE result is contamination-free. I treat it as later consistency with the ordering, not as equally clean validation.

That caveat is healthy. A benchmark that once separated the frontier will eventually start to saturate as the frontier catches it. DeepSWE earns trust by making the tasks and trajectories inspectable, not by promising to remain the final leaderboard forever.

Success rate without cost is half a decision

The more useful part of the live leaderboard may be the chart beside the score. DeepSWE reports average API cost, output tokens, and agent steps per task. That lets us stop asking only “which configuration wins?” and ask “what am I paying for the next point of reliability?”

The all-effort artifact was generated on July 17, 2026. I used GPT-5.5 xhigh’s 67.0% pass@1 as the floor, then removed any configuration for which another result had equal-or-better pass@1 and pass@4 at equal-or-lower average cost, with at least one strict advantage. That leaves the high-success slice of the three-metric Pareto frontier.

Pass@1 measures reliability on one rollout. Pass@4 asks whether at least one of roughly four rollouts solved the task. It is measured directly; it should not be reconstructed by pretending retries are independent, because difficult tasks tend to remain difficult. The cost column is the mean for one scored rollout. Four unconditional rollouts would cost roughly four times the displayed amount; a retry policy that stops after success would cost less.

The interesting comparison is not simply OpenAI versus Anthropic. It is Sol versus Terra versus Luna, and one reasoning level against another. Sol at xhigh gives up only two points relative to Sol at max, while max costs roughly 78% more. Sol at high still reaches 69.4% for $3.47. Luna gives up about five and a half points relative to Sol at max while costing roughly 64% less per task. Max buys the highest one-shot success rate, but the cheaper settings may buy more completed work per dollar.

Fable shows why the selection rule matters. Its max setting is absent because xhigh dominates it on the current point estimates: 0.2 points higher pass@1, 4.4 points higher pass@4, and 38% lower average cost. That does not prove xhigh is intrinsically better—these are noisy samples—but max is the wrong representative row for an efficiency comparison.

Around 70%, Terra max and Fable xhigh make the cleaner comparison. Their pass@1 estimates differ by 0.3 points, and both reach 88.5% pass@4. Their average rollout costs are $4.95 and $13.41. Terra is 63% cheaper. Their pass@1 confidence intervals overlap; the invoices do not.

Pass@4 adds another wrinkle. Kimi K3 reaches 68.5% pass@1 and 89.4% pass@4 for $4.65. Luna has a lower one-shot result than Sol at max, but a higher pass@4: 90.3% against 85.8%. That does not make Luna universally better. It says that dependable performance on one attempt and reachability across several attempts are different properties. A workflow that can cheaply retry may choose differently from one that needs an expensive first run to succeed.

On the displayed point estimates, this is a high-success slice of a three-metric Pareto frontier, not a production ranking. If a failure creates an hour of senior review or risks a production incident, paying for the highest success rate can be cheap. If I am running many bounded changes with strong automated checks, a lower-cost configuration may be the rational choice. The token invoice is only one cost; failed work, review time, latency, and cleanup are usually larger.

What DeepSWE still does not measure

DeepSWE is superior for the question I care about, not for every possible question about coding agents.

• It grades functional correctness, not readability, architecture, documentation, restraint, or whether a maintainer would accept the patch without revision.
• Its reward is binary. Missing one edge case and failing to compile both score zero, even though they create very different cleanup work.
• Its shared bash-only harness compares models under a controlled loop, not the native products people use every day.
• It concentrates on long-horizon implementation in established open-source repositories. Planning, review, interactive debugging, underspecified requests, proprietary systems, and several major languages remain outside the frame.
• Its qualitative verifier audit used GPT-5.5 as the LLM judge. The authors disclose that self-preference cannot be excluded, so I treat the reported family-specific failure patterns as hypotheses, not verdicts.

None of that invalidates the result. It tells us what the result means. The best benchmark is not the one with the grandest claim; it is the one whose boundary is clear enough that you know when to trust it.

A benchmark should survive contact with the work

I do not trust DeepSWE because GPT models score well on it. I trust it more because its task construction makes sense, its artifacts can be inspected, and its first ordering matched an opinion I had formed before seeing the chart.

For long-horizon coding-agent work, it is currently the most useful public benchmark I have seen. It tells a believable story about which models finish the work, how they fail, and what each attempt costs. That is much closer to the decision I actually need to make than a compressed leaderboard built from yesterday’s merged patches.

The next benchmark I trust may disagree with me. It should. The point is not to find a chart that confirms a favorite model. The point is to find a measurement strong enough to change which model I use.