Metric
resolved (%)
Status
Saturating
Category
Coding
Maintainer
Princeton & Stanford; Verified by OpenAI; curated by Epoch AI
Chance baseline
0%
What it measures
SWE-bench Verified measures whether an AI system can resolve real software engineering issues in real codebases. Each task gives a model an actual bug report or feature request drawn from a popular open-source Python project, along with the project's full code, and asks it to produce a code change that fixes the issue. It is the human-validated subset of SWE-bench, the benchmark introduced in the 2023 paper "SWE-bench: Can Language Models Resolve Real-World GitHub Issues?" by Carlos Jimenez, John Yang, and collaborators at Princeton University and partner institutions. Where earlier coding benchmarks tested isolated function writing, SWE-bench measures end-to-end engineering: understanding an issue, navigating a large repository, and editing the right lines across multiple files.
The task in detail
The original SWE-bench comprises 2,294 task instances built from real GitHub issues and the pull requests that closed them, across 12 widely used Python repositories. Each instance pins the repository to the commit just before the fix, provides the issue text as the problem statement, and hides the human pull request that actually resolved it. To succeed, a model (usually wrapped in an agent that can read files, search, and run commands) must generate a patch that edits the codebase. Resolving an issue frequently requires coordinating changes across multiple functions, classes, and files at once, which is what makes the task realistic and hard. SWE-bench Verified keeps this exact task format but restricts it to 500 instances that human engineers confirmed are well-specified and fairly testable.
How models are scored
Scoring is execution-based, not a similarity judgment. After a model's patch is applied to the repository, the harness runs the project's own unit tests. Each instance defines two test sets: FAIL_TO_PASS tests, which fail before the fix and must pass after it, and PASS_TO_PASS tests, which must stay passing so the change does not break existing behavior. An instance counts as resolved only if all of the required tests pass. The headline metric is the percentage of instances resolved. Each dataset instance carries the fields needed to run this check, including the repository, the base commit, the problem statement, the gold (human) patch, and the test patch that supplies the tests.
How to read a score
A SWE-bench Verified score is the share of 500 real issues a system fully fixes, so it maps intuitively to "how often could this agent close a real ticket." It is stricter than it sounds, because partial fixes score nothing when the tests must all pass. Two things shape interpretation. First, results depend heavily on the surrounding agent scaffold (how the model explores the repo and runs tests), not just the base model, so a model name alone does not determine the score. Second, because the tests, not a human, decide success, a passing patch is one the tests accept, which may differ from the exact human fix. Comparisons are only fair within the same benchmark version and evaluation harness.
History and notable results
The original SWE-bench appeared in October 2023 and was published at ICLR 2024. At the time it was brutally hard: the best system in the paper, using Claude 2, resolved just 1.96% of issues. In August 2024, OpenAI released SWE-bench Verified in collaboration with the SWE-bench authors to fix flaws that made the original understate model ability. A team of 93 experienced Python developers annotated 1,699 randomly sampled test instances, flagging about 38.3% for underspecified problem statements and about 61.1% for unit tests that might mark a valid solution wrong. Reviewing coding tasks this closely is now paid hourly work for experienced engineers, and AITraining.jobs' roles for software engineers, from a jobs board we also built, lists the open ones and what each platform pays. After filtering, they assembled a clean 500-problem set and rated each by estimated time to fix. On this subset, OpenAI reported that GPT-4o resolved 33.2% of issues, more than double its roughly 16% on the original. Scores rose quickly afterward, and by 2025 the strongest coding agents resolved more than half of the set, so the benchmark is now widely considered near saturation for frontier systems.
Limitations and criticism
SWE-bench Verified fixed specific problems in the original (bad tests, unclear issues) but keeps structural limits. It is Python-only and drawn from 12 repositories, so it does not represent other languages, smaller projects, or greenfield work. Because the issues and their fixes are public on GitHub and predate model training cutoffs, contamination is a serious concern: a model may have seen the actual patch during pretraining. Public code is already in training sets, which is why code that was never public (private repositories with their commits, reviews and tickets) is bought for training on its own, at the per-repository prices listed on AIData.Broker, a companion index to this site, each with its source. A related risk is solution leakage: because each fix already exists publicly, an agent that can reach the web or the repository's own history might recover hints toward the human patch instead of reasoning it out, so careful evaluations sandbox the environment. Even with good tests, passing the provided unit tests is only a proxy for correctness: a patch can satisfy the tests without being the intended fix, and the tests may not cover every behavior. Independent evaluators handle these edges differently. Epoch AI, for example, runs 484 of the 500 instances (excluding 16 that do not run reliably on its infrastructure), uses three annotators per sample, and estimates a residual error rate of roughly 5 to 10%.
Related benchmarks
- SWE-bench (original): the full 2,294-instance set the Verified subset is drawn from.
- SWE-bench Lite, Multimodal, and Multilingual: official variants that trim the set for cheaper evaluation or extend it beyond text and beyond Python.
- Function-level code benchmarks such as HumanEval and MBPP: earlier, much narrower tests of writing a single correct function, which SWE-bench was designed to move beyond.
References
Leaderboard
Checked Oct 30, 2026, 07:00 UTC (refresh due) · scores last changed Oct 10, 2026Best SWE-bench Verified score per model, ranked. Each score links to its source.
| # | Model | Score | Source |
|---|---|---|---|
| 1 | Anthropic: Claude Opus 4.7(max)Anthropic | 83.5% | epoch |
| 2 | gpt-5.5-pre-release_xhigh(xhigh)OpenAI | 80.6% | epoch |
| 3 | gpt-5.5-pre-release_xhigh(xhigh)OpenAI | 80.6% | epoch |
| 4 | Google: Gemini 3.5 Flash(high)Google DeepMind | 79.3% | epoch |
| 5 | Anthropic: Claude Opus 4.6Anthropic | 78.7% | epoch |
| 6 | Z.ai: GLM 5.2(max)Z.ai (Zhipu AI) | 78.7% | epoch |
| 7 | DeepSeek: DeepSeek V4 Pro 0423(max)DeepSeek | 77.6% | epoch |
| 8 | Qwen: Qwen3.7 MaxAlibaba | 77.3% | epoch |
| 9 | OpenAI: GPT-5.4(high)OpenAI | 76.9% | epoch |
| 10 | qwen3.6-max-previewAlibaba | 76.7% | epoch |
| 11 | qwen3.6-max-previewAlibaba | 76.7% | epoch |
| 12 | MoonshotAI: Kimi K2.6Moonshot | 76.7% | epoch |
| 13 | claude-opus-4-5-20251101Anthropic | 76.7% | epoch |
| 14 | Google: Gemini 3.1 Pro Preview Custom ToolsGoogle DeepMind | 75.6% | epoch |
| 15 | Google: Gemini 3 Flash PreviewGoogle DeepMind | 75.4% | epoch |
| 16 | Anthropic: Claude Sonnet 4.6Anthropic | 75.2% | epoch |
| 17 | OpenAI: GPT-5.3-Codex(high)OpenAI | 74.8% | epoch |
| 18 | Z.ai: GLM 5.1Z.ai (Zhipu AI) | 74.2% | epoch |
| 19 | MoonshotAI: Kimi K2.5Moonshot | 73.8% | epoch |
| 20 | OpenAI: GPT-5.2(high)OpenAI | 73.8% | epoch |
| 21 | OpenAI: GPT-5(high)OpenAI | 73.6% | epoch |
| 22 | Anthropic: Claude Opus 4.1Anthropic | 73.3% | epoch |
| 23 | gemini-3-pro-previewGoogle DeepMind | 72.9% | epoch |
| 24 | gemini-3-pro-previewGoogle DeepMind | 72.9% | epoch |
| 25 | Z.ai: GLM 5Z.ai (Zhipu AI) | 72.1% | epoch |
| 26 | Anthropic: Claude Sonnet 4.5Anthropic | 71.3% | epoch |
| 27 | claude-opus-4-20250514Anthropic | 70.7% | epoch |
| 28 | OpenAI: GPT-5.1(high)OpenAI | 68% | epoch |
| 29 | OpenAI: GPT-5 Mini(medium)OpenAI | 64.7% | epoch |
| 30 | OpenAI: o3(medium)OpenAI | 62.3% | epoch |
| 31 | claude-3-7-sonnet-20250219Anthropic | 61% | epoch |
| 32 | claude-3-7-sonnet-20250219Anthropic | 61% | epoch |
| 33 | Qwen: Qwen3.6 PlusAlibaba | 57.9% | epoch |
| 34 | Google: Gemini 2.5 ProGoogle DeepMind | 57.6% | epoch |
| 35 | OpenAI: GPT-4.1OpenAI | 48.5% | epoch |
| 36 | OpenAI: GPT-4o (2024-11-20)OpenAI | 31% | epoch |
Run your own prompt on the top 3
Claude Opus 4.7, Gemini 3.5 Flash, Claude Opus 4.6, side by side on the same prompt, with each one's time and cost. A free account includes starter credit.
Compare the leaders
In our guides
The guides that explain this benchmark or show its live leaderboard.
More Coding benchmarks
Aider Polyglot
Editing code correctly across six languages, including producing valid diffs.
SciCode
Research-level scientific computing curated by scientists.
Terminal-Bench
End-to-end agentic tasks in a real sandboxed terminal.
LiveBench
A contamination-resistant, frequently-refreshed general benchmark.
WeirdML
Unusual machine-learning problems solved by writing and iterating on real code.
EHRSQL
Translating clinical data questions into executable SQL over an electronic health-record schema.
CadEval
Turning a written description into a geometrically correct CAD model.
DeepSWE
Solving original, sustained software-engineering tasks across real codebases.
ExploitBench
Measuring progress through graded software-exploitation milestones in a controlled evaluation.
FrontierCode
Producing repository changes that satisfy a maintainer's standards for merging.
GSO OPT@1
Optimizing software to approach expert speedups while preserving correctness.
MirrorCode (ML, +Private, 2L)
Reimplementing entire programs from observable behavior without their source code.
Surface Evolver Bench
Building physically consistent liquid-surface simulations in a specialized language.
SWE-bench Verified is maintained by Princeton & Stanford; Verified by OpenAI; curated by Epoch AI. Scores on this page are collected from openly-licensed sources and each links to its origin. See the full matrix to compare every model across every benchmark.