AI benchmarks and leaderboards are different things
Learn the difference between a benchmark, an evaluation run and a leaderboard, and use each to answer the right question.
LLM Stats
SWE-Bench Pro
SWE-Bench Pro is an advanced version of SWE-Bench that evaluates language models on complex, real-world software engineering tasks requiring extended reasoning and multi-step problem solving.
Evaluation context and evidence
- Source
- LLM Stats
- Publisher
- Scale AI
- Benchmark release
- 2025-09-05
- Collected
- 2026-08-17
- Modality
- text
These are source-reported observations, not independent reruns by this site. Check the task version, evaluation protocol, unit and reporting date before comparing models. Results from different setups are not a universal ranking.
Original sources
Reported results 50
Shown in source observation order, not ranked. Dates retain the precision provided by the source.
| Model | Value | Unit | Protocol | Reported date and basis | Source |
|---|---|---|---|---|---|
| Claude Fable 5 | 0.8 | Not recorded | Not recorded | 2026-06-09 · model_announcement | LLM Stats |
| Claude Mythos Preview | 0.778 | Not recorded | Not recorded | 2026-04-07 · model_announcement | LLM Stats |
| Claude Opus 4.8 | 0.692 | Not recorded | Not recorded | 2026-05-28 · model_announcement | LLM Stats |
| Qwen3.8 Max | 0.677 | Not recorded | Not recorded | 2026-08-02 · model_announcement | LLM Stats |
| Grok 4.5 | 0.647 | Not recorded | Not recorded | 2026-07-16 · model_announcement | LLM Stats |
| GPT-5.6 Sol | 0.646 | Not recorded | Not recorded | 2026-07-09 · model_announcement | LLM Stats |
| Claude Opus 4.7 | 0.643 | Not recorded | Not recorded | 2026-04-16 · model_announcement | LLM Stats |
| GPT-5.6 Terra | 0.634 | Not recorded | Not recorded | 2026-07-09 · model_announcement | LLM Stats |
| Claude Sonnet 5 | 0.632 | Not recorded | Not recorded | 2026-06-30 · model_announcement | LLM Stats |
| GPT-5.6 Luna | 0.627 | Not recorded | Not recorded | 2026-07-09 · model_announcement | LLM Stats |
| GLM-5.2 | 0.621 | Not recorded | Not recorded | 2026-06-16 · model_announcement | LLM Stats |
| Qwen3.8-27B | 0.617 | Not recorded | Not recorded | 2026-08-14 · model_announcement | LLM Stats |
| Muse Spark 1.1 | 0.615 | Not recorded | Not recorded | 2026-07-09 · model_announcement | LLM Stats |
| Qwen3.7 Max | 0.606 | Not recorded | Not recorded | 2026-05-19 · model_announcement | LLM Stats |
| Laguna S 2.1 | 0.594 | Not recorded | Not recorded | 2026-07-21 · model_announcement | LLM Stats |
| MiniMax M3 | 0.59 | Not recorded | Not recorded | 2026-06-01 · model_announcement | LLM Stats |
| Gemini 3.6 Flash | 0.587 | Not recorded | Not recorded | 2026-07-21 · model_announcement | LLM Stats |
| GPT-5.5 | 0.586 | Not recorded | Not recorded | 2026-04-23 · model_announcement | LLM Stats |
| Kimi K2.6 | 0.586 | Not recorded | Not recorded | 2026-04-20 · model_announcement | LLM Stats |
| GLM-5.1 | 0.584 | Not recorded | Not recorded | 2026-04-07 · model_announcement | LLM Stats |
| Hy3 | 0.579 | Not recorded | Not recorded | 2026-07-06 · model_announcement | LLM Stats |
| GPT-5.4 | 0.577 | Not recorded | Not recorded | 2026-03-05 · model_announcement | LLM Stats |
| Qwen3.7-Plus | 0.576 | Not recorded | Not recorded | 2026-05-31 · model_announcement | LLM Stats |
| Seed 2.1 Pro | 0.575 | Not recorded | Not recorded | 2026-06-24 · model_announcement | LLM Stats |
| MiMo-V2.5-Pro | 0.572 | Not recorded | Not recorded | 2026-04-27 · model_announcement | LLM Stats |
| Seed 2.1 Turbo | 0.57 | Not recorded | Not recorded | 2026-06-24 · model_announcement | LLM Stats |
| GPT-5.3 Codex | 0.568 | Not recorded | Not recorded | 2026-02-05 · model_announcement | LLM Stats |
| Qwen3.6 Plus | 0.566 | Not recorded | Not recorded | 2026-04-02 · model_announcement | LLM Stats |
| GPT-5.2 Codex | 0.564 | Not recorded | Not recorded | 2026-01-14 · model_announcement | LLM Stats |
| MiniMax M2.7 | 0.562 | Not recorded | Not recorded | 2026-03-18 · model_announcement | LLM Stats |
| MiMo-V2.5 | 0.561 | Not recorded | Not recorded | 2026-04-22 · model_announcement | LLM Stats |
| Inkling-Small | 0.559 | Not recorded | Not recorded | 2026-07-30 · model_announcement | LLM Stats |
| DeepSeek-V4-Pro-Max | 0.554 | Not recorded | Not recorded | 2026-04-23 · model_announcement | LLM Stats |
| MiniMax M2.5 | 0.554 | Not recorded | Not recorded | 2026-02-12 · model_announcement | LLM Stats |
| Gemini 3.5 Flash | 0.551 | Not recorded | Not recorded | 2026-05-19 · model_announcement | LLM Stats |
| GPT-5.4 mini | 0.544 | Not recorded | Not recorded | 2026-03-17 · model_announcement | LLM Stats |
| Gemini 3.5 Flash-Lite | 0.542 | Not recorded | Not recorded | 2026-07-21 · model_announcement | LLM Stats |
| Gemini 3.1 Pro | 0.542 | Not recorded | Not recorded | 2026-02-19 · model_announcement | LLM Stats |
| Qwen3.6-27B | 0.535 | Not recorded | Not recorded | 2026-04-21 · model_announcement | LLM Stats |
| MAI-Thinking-1 | 0.528 | Not recorded | Not recorded | 2026-06-02 · model_announcement | LLM Stats |
| DeepSeek-V4-Flash-Max | 0.526 | Not recorded | Not recorded | 2026-04-23 · model_announcement | LLM Stats |
| Muse Spark | 0.524 | Not recorded | Not recorded | 2026-04-08 · model_announcement | LLM Stats |
| GPT-5.4 nano | 0.524 | Not recorded | Not recorded | 2026-03-17 · model_announcement | LLM Stats |
| DeepSeek-V4-Flash-0423 | 0.523 | Not recorded | Not recorded | 2026-04-23 · model_announcement | LLM Stats |
| Muse Glimmer-30B | 0.512 | Not recorded | Not recorded | 2026-08-10 · model_announcement | LLM Stats |
| MAI-Code-1-Flash | 0.512 | Not recorded | Not recorded | 2026-06-02 · model_announcement | LLM Stats |
| Kimi K2.5 | 0.507 | Not recorded | Not recorded | 2026-01-27 · model_announcement | LLM Stats |
| Qwen3.6-35B-A3B | 0.495 | Not recorded | Not recorded | 2026-04-16 · model_announcement | LLM Stats |
| Laguna XS 2.1 | 0.476 | Not recorded | Not recorded | 2026-07-02 · model_announcement | LLM Stats |
| North Mini Code 1.0 | 0.402 | Not recorded | Not recorded | 2026-06-09 · model_announcement | LLM Stats |