AI benchmarks and leaderboards are different things
Learn the difference between a benchmark, an evaluation run and a leaderboard, and use each to answer the right question.
LLM Stats
SWE-Bench Verified
A verified subset of 500 software engineering problems from real GitHub issues, validated by human annotators for evaluating language models' ability to resolve real-world coding issues by generating patches for Python codebases.
Evaluation context and evidence
- Source
- LLM Stats
- Publisher
- OpenAI
- Benchmark release
- 2024-08-13
- Collected
- 2026-08-17
- Modality
- text
These are source-reported observations, not independent reruns by this site. Check the task version, evaluation protocol, unit and reporting date before comparing models. Results from different setups are not a universal ranking.
Original sources
Reported results 111
Shown in source observation order, not ranked. Dates retain the precision provided by the source.
| Model | Value | Unit | Protocol | Reported date and basis | Source |
|---|---|---|---|---|---|
| Claude Fable 5 | 0.95 | Not recorded | Not recorded | 2026-06-09 · model_announcement | LLM Stats |
| Claude Mythos Preview | 0.939 | Not recorded | Not recorded | 2026-04-07 · model_announcement | LLM Stats |
| Claude Opus 4.8 | 0.886 | Not recorded | Not recorded | 2026-05-28 · model_announcement | LLM Stats |
| Claude Opus 4.7 | 0.876 | Not recorded | Not recorded | 2026-04-16 · model_announcement | LLM Stats |
| Claude Sonnet 5 | 0.852 | Not recorded | Not recorded | 2026-06-30 · model_announcement | LLM Stats |
| Claude Opus 4.5 | 0.809 | Not recorded | Not recorded | 2025-11-24 · model_announcement | LLM Stats |
| Claude Opus 4.6 | 0.808 | Not recorded | Not recorded | 2026-02-05 · model_announcement | LLM Stats |
| DeepSeek-V4-Pro-Max | 0.806 | Not recorded | Not recorded | 2026-04-23 · model_announcement | LLM Stats |
| Gemini 3.1 Pro | 0.806 | Not recorded | Not recorded | 2026-02-19 · model_announcement | LLM Stats |
| MiniMax M3 | 0.805 | Not recorded | Not recorded | 2026-06-01 · model_announcement | LLM Stats |
| Qwen3.7 Max | 0.804 | Not recorded | Not recorded | 2026-05-19 · model_announcement | LLM Stats |
| Inkling-Small | 0.802 | Not recorded | Not recorded | 2026-07-30 · model_announcement | LLM Stats |
| Kimi K2.6 | 0.802 | Not recorded | Not recorded | 2026-04-20 · model_announcement | LLM Stats |
| MiniMax M2.5 | 0.802 | Not recorded | Not recorded | 2026-02-12 · model_announcement | LLM Stats |
| GPT-5.2 | 0.8 | Not recorded | Not recorded | 2025-12-11 · model_announcement | LLM Stats |
| Claude Sonnet 4.6 | 0.796 | Not recorded | Not recorded | 2026-02-17 · model_announcement | LLM Stats |
| DeepSeek-V4-Flash-Max | 0.79 | Not recorded | Not recorded | 2026-04-23 · model_announcement | LLM Stats |
| MiMo-V2.5-Pro | 0.789 | Not recorded | Not recorded | 2026-04-27 · model_announcement | LLM Stats |
| Qwen3.6 Plus | 0.788 | Not recorded | Not recorded | 2026-04-02 · model_announcement | LLM Stats |
| DeepSeek-V4-Flash-0423 | 0.786 | Not recorded | Not recorded | 2026-04-23 · model_announcement | LLM Stats |
| Hy3 | 0.78 | Not recorded | Not recorded | 2026-07-06 · model_announcement | LLM Stats |
| MiMo-V2-Pro | 0.78 | Not recorded | Not recorded | 2026-03-18 · model_announcement | LLM Stats |
| Gemini 3 Flash | 0.78 | Not recorded | Not recorded | 2025-12-17 · model_announcement | LLM Stats |
| GLM-5 | 0.778 | Not recorded | Not recorded | 2026-02-11 · model_announcement | LLM Stats |
| Qwen3.7-Plus | 0.777 | Not recorded | Not recorded | 2026-05-31 · model_announcement | LLM Stats |
| Mistral Medium 3.5 | 0.776 | Not recorded | Not recorded | 2026-04-29 · model_announcement | LLM Stats |
| Muse Spark | 0.774 | Not recorded | Not recorded | 2026-04-08 · model_announcement | LLM Stats |
| Qwen3.6-27B | 0.772 | Not recorded | Not recorded | 2026-04-21 · model_announcement | LLM Stats |
| Kimi K2.5 | 0.768 | Not recorded | Not recorded | 2026-01-27 · model_announcement | LLM Stats |
| Seed 2.0 Pro | 0.765 | Not recorded | Not recorded | 2026-02-14 · model_announcement | LLM Stats |
| Qwen3.5-397B-A17B | 0.764 | Not recorded | Not recorded | 2026-02-16 · model_announcement | LLM Stats |
| GPT-5.1 | 0.763 | Not recorded | Not recorded | 2025-11-13 · model_announcement | LLM Stats |
| GPT-5.1 Instant | 0.763 | Not recorded | Not recorded | 2025-11-12 · model_announcement | LLM Stats |
| GPT-5.1 Thinking | 0.763 | Not recorded | Not recorded | 2025-11-12 · model_announcement | LLM Stats |
| Gemini 3 Pro | 0.762 | Not recorded | Not recorded | 2025-11-18 · model_announcement | LLM Stats |
| Muse Glimmer-30B | 0.76 | Not recorded | Not recorded | 2026-08-10 · model_announcement | LLM Stats |
| GPT-5 | 0.749 | Not recorded | Not recorded | 2025-08-07 · model_announcement | LLM Stats |
| MiMo-V2-Omni | 0.748 | Not recorded | Not recorded | 2026-03-18 · model_announcement | LLM Stats |
| GPT-5 Codex | 0.745 | Not recorded | Not recorded | 2025-09-15 · model_announcement | LLM Stats |
| Claude Opus 4.1 | 0.745 | Not recorded | Not recorded | 2025-08-05 · model_announcement | LLM Stats |
| Step-3.5-Flash | 0.744 | Not recorded | Not recorded | 2026-02-02 · model_announcement | LLM Stats |
| GLM-4.7 | 0.738 | Not recorded | Not recorded | 2025-12-22 · model_announcement | LLM Stats |
| GPT-5.1 Codex | 0.737 | Not recorded | Not recorded | 2025-11-19 · model_announcement | LLM Stats |
| MAI-Thinking-1 | 0.735 | Not recorded | Not recorded | 2026-06-02 · model_announcement | LLM Stats |
| Seed 2.0 Lite | 0.735 | Not recorded | Not recorded | 2026-02-14 · model_announcement | LLM Stats |
| Qwen3.6-35B-A3B | 0.734 | Not recorded | Not recorded | 2026-04-16 · model_announcement | LLM Stats |
| MiMo-V2-Flash | 0.734 | Not recorded | Not recorded | 2025-12-16 · model_announcement | LLM Stats |
| Claude Haiku 4.5 | 0.733 | Not recorded | Not recorded | 2025-10-15 · model_announcement | LLM Stats |
| DeepSeek-V3.2-Speciale | 0.731 | Not recorded | Not recorded | 2025-12-01 · model_announcement | LLM Stats |
| DeepSeek-V3.2 | 0.731 | Not recorded | Not recorded | 2025-12-01 · model_announcement | LLM Stats |
| DeepSeek-V3.2 (Thinking) | 0.731 | Not recorded | Not recorded | 2025-12-01 · model_announcement | LLM Stats |
| Claude Sonnet 4 | 0.727 | Not recorded | Not recorded | 2025-05-22 · model_announcement | LLM Stats |
| MAI-Code-1.1-Flash | 0.726 | Not recorded | Not recorded | 2026-08-11 · model_announcement | LLM Stats |
| Claude Opus 4 | 0.725 | Not recorded | Not recorded | 2025-05-22 · model_announcement | LLM Stats |
| Qwen3.5-27B | 0.724 | Not recorded | Not recorded | 2026-02-24 · model_announcement | LLM Stats |
| Qwen3.5-122B-A10B | 0.72 | Not recorded | Not recorded | 2026-02-24 · model_announcement | LLM Stats |
| MAI-Code-1-Flash | 0.716 | Not recorded | Not recorded | 2026-06-02 · model_announcement | LLM Stats |
| Kimi K2-Thinking-0905 | 0.713 | Not recorded | Not recorded | 2025-09-05 · model_announcement | LLM Stats |
| Laguna XS 2.1 | 0.709 | Not recorded | Not recorded | 2026-07-02 · model_announcement | LLM Stats |
| Grok Code Fast 1 | 0.708 | Not recorded | Not recorded | 2025-08-28 · model_announcement | LLM Stats |
| Nemotron 3 Ultra (550B A55B) | 0.707 | Not recorded | Not recorded | 2026-06-04 · model_announcement | LLM Stats |
| Solar Pro 4 | 0.706 | Not recorded | Not recorded | 2026-08-06 · model_announcement | LLM Stats |
| Claude 3.7 Sonnet | 0.703 | Not recorded | Not recorded | 2025-02-24 · model_announcement | LLM Stats |
| LongCat-Flash-Thinking-2601 | 0.7 | Not recorded | Not recorded | 2026-01-14 · model_announcement | LLM Stats |
| Nova 2 Pro | 0.7 | Not recorded | Not recorded | 2025-12-02 · model_announcement | LLM Stats |
| Qwen3 Max | 0.696 | Not recorded | Not recorded | 2025-12-15 · model_announcement | LLM Stats |
| Qwen3-Coder 480B A35B Instruct | 0.696 | Not recorded | Not recorded | 2025-01-31 · model_announcement | LLM Stats |
| MiniMax M2 | 0.694 | Not recorded | Not recorded | 2025-10-27 · model_announcement | LLM Stats |
| Qwen3.5-35B-A3B | 0.692 | Not recorded | Not recorded | 2026-02-24 · model_announcement | LLM Stats |
| o3 | 0.691 | Not recorded | Not recorded | 2025-04-16 · model_announcement | LLM Stats |
| o4-mini | 0.681 | Not recorded | Not recorded | 2025-04-16 · model_announcement | LLM Stats |
| GLM-4.6 | 0.68 | Not recorded | Not recorded | 2025-09-30 · model_announcement | LLM Stats |
| DeepSeek-V3.2-Exp | 0.678 | Not recorded | Not recorded | 2025-09-29 · model_announcement | LLM Stats |
| North Mini Code 1.0 | 0.676 | Not recorded | Not recorded | 2026-06-09 · model_announcement | LLM Stats |
| Gemini 2.5 Pro Preview 06-05 | 0.672 | Not recorded | Not recorded | 2025-06-05 · model_announcement | LLM Stats |
| MiniMax M2.1 | 0.67 | Not recorded | Not recorded | 2025-12-23 · model_announcement | LLM Stats |
| DeepSeek-V3.1 | 0.66 | Not recorded | Not recorded | 2025-01-10 · model_announcement | LLM Stats |
| Kimi K2-Instruct-0905 | 0.658 | Not recorded | Not recorded | 2025-09-05 · model_announcement | LLM Stats |
| Nova 2 Lite | 0.645 | Not recorded | Not recorded | 2025-12-02 · model_announcement | LLM Stats |
| GLM-4.5 | 0.642 | Not recorded | Not recorded | 2025-07-28 · model_announcement | LLM Stats |
| Gemini 2.5 Pro | 0.632 | Not recorded | Not recorded | 2025-05-20 · model_announcement | LLM Stats |
| Devstral Medium | 0.616 | Not recorded | Not recorded | 2025-07-10 · model_announcement | LLM Stats |
| LongCat-Flash-Chat | 0.604 | Not recorded | Not recorded | 2025-08-29 · model_announcement | LLM Stats |
| Gemini 2.5 Flash | 0.604 | Not recorded | Not recorded | 2025-05-20 · model_announcement | LLM Stats |
| LongCat-Flash-Thinking | 0.594 | Not recorded | Not recorded | 2025-09-22 · model_announcement | LLM Stats |
| GLM-4.7-Flash | 0.592 | Not recorded | Not recorded | 2026-01-19 · model_announcement | LLM Stats |
| GLM-4.5-Air | 0.576 | Not recorded | Not recorded | 2025-07-28 · model_announcement | LLM Stats |
| MiniMax M1 80K | 0.56 | Not recorded | Not recorded | 2025-06-16 · model_announcement | LLM Stats |
| MiniMax M1 40K | 0.556 | Not recorded | Not recorded | 2025-06-16 · model_announcement | LLM Stats |
| GPT-4.1 | 0.546 | Not recorded | Not recorded | 2025-04-14 · model_announcement | LLM Stats |
| LongCat-Flash-Lite | 0.544 | Not recorded | Not recorded | 2026-02-05 · model_announcement | LLM Stats |
| Nemotron 3 Super (120B A12B) | 0.5373 | Not recorded | Not recorded | 2026-03-11 · model_announcement | LLM Stats |
| Devstral Small 1.1 | 0.536 | Not recorded | Not recorded | 2025-07-11 · model_announcement | LLM Stats |
| Nemotron 3.5 Lightning (30B A3B) | 0.5156 | Not recorded | Not recorded | 2026-08-11 · model_announcement | LLM Stats |
| o3-mini | 0.493 | Not recorded | Not recorded | 2025-01-30 · model_announcement | LLM Stats |
| Claude 3.5 Sonnet | 0.49 | Not recorded | Not recorded | 2024-10-22 · model_announcement | LLM Stats |
| Sarvam-105B | 0.45 | Not recorded | Not recorded | 2026-03-06 · model_announcement | LLM Stats |
| DeepSeek-R1-0528 | 0.446 | Not recorded | Not recorded | 2025-05-28 · model_announcement | LLM Stats |
| DeepSeek-V3 | 0.42 | Not recorded | Not recorded | 2024-12-25 · model_announcement | LLM Stats |
| o1-preview | 0.413 | Not recorded | Not recorded | 2024-09-12 · model_announcement | LLM Stats |
| o1 | 0.41 | Not recorded | Not recorded | 2024-12-17 · model_announcement | LLM Stats |
| Claude 3.5 Haiku | 0.406 | Not recorded | Not recorded | 2024-10-22 · model_announcement | LLM Stats |
| Nemotron 3 Nano (30B A3B) | 0.388 | Not recorded | Not recorded | 2025-12-15 · model_announcement | LLM Stats |
| GPT-4.5 | 0.38 | Not recorded | Not recorded | 2025-02-27 · model_announcement | LLM Stats |
| Sarvam-30B | 0.34 | Not recorded | Not recorded | 2026-03-06 · model_announcement | LLM Stats |
| GPT-4o | 0.332 | Not recorded | Not recorded | 2024-08-06 · model_announcement | LLM Stats |
| Gemini 2.5 Flash-Lite | 0.316 | Not recorded | Not recorded | 2025-06-17 · model_announcement | LLM Stats |
| GPT-4.1 mini | 0.236 | Not recorded | Not recorded | 2025-04-14 · model_announcement | LLM Stats |
| Gemini Diffusion | 0.229 | Not recorded | Not recorded | 2025-05-20 · model_announcement | LLM Stats |
| DeepSeek-V2.5 | 0.168 | Not recorded | Not recorded | 2024-05-08 · model_announcement | LLM Stats |
| GPT-4o mini | 0.087 | Not recorded | Not recorded | 2024-07-18 · model_announcement | LLM Stats |