AI benchmarks and leaderboards are different things
Learn the difference between a benchmark, an evaluation run and a leaderboard, and use each to answer the right question.
LLM Stats
MMLU-Pro
A more robust and challenging multi-task language understanding benchmark that extends MMLU by expanding multiple-choice options from 4 to 10, eliminating trivial questions, and focusing on reasoning-intensive tasks. Features over 12,000 curated questions across 14 domains and causes a 16-33% accuracy drop compared to original MMLU.
Evaluation context and evidence
- Source
- LLM Stats
- Publisher
- University of Waterloo
- Benchmark release
- 2024-06-03
- Collected
- 2026-08-17
- Modality
- text
These are source-reported observations, not independent reruns by this site. Check the task version, evaluation protocol, unit and reporting date before comparing models. Results from different setups are not a universal ranking.
Original sources
Reported results 134
Shown in source observation order, not ranked. Dates retain the precision provided by the source.
| Model | Value | Unit | Protocol | Reported date and basis | Source |
|---|---|---|---|---|---|
| Sakana Namazu | 0.9033 | Not recorded | Not recorded | 2026-08-03 · model_announcement | LLM Stats |
| Qwen3.7 Max | 0.896 | Not recorded | Not recorded | 2026-05-19 · model_announcement | LLM Stats |
| Qwen3.7-Plus | 0.885 | Not recorded | Not recorded | 2026-05-31 · model_announcement | LLM Stats |
| Qwen3.6 Plus | 0.885 | Not recorded | Not recorded | 2026-04-02 · model_announcement | LLM Stats |
| MiniMax M2.1 | 0.88 | Not recorded | Not recorded | 2025-12-23 · model_announcement | LLM Stats |
| Qwen3.5-397B-A17B | 0.878 | Not recorded | Not recorded | 2026-02-16 · model_announcement | LLM Stats |
| DeepSeek-V4-Pro-Max | 0.875 | Not recorded | Not recorded | 2026-04-23 · model_announcement | LLM Stats |
| Kimi K2.5 | 0.871 | Not recorded | Not recorded | 2026-01-27 · model_announcement | LLM Stats |
| ERNIE 5.0 | 0.87 | Not recorded | Not recorded | 2026-01-22 · model_announcement | LLM Stats |
| Nemotron 3 Ultra (550B A55B) | 0.868 | Not recorded | Not recorded | 2026-06-04 · model_announcement | LLM Stats |
| Qwen3.5-122B-A10B | 0.867 | Not recorded | Not recorded | 2026-02-24 · model_announcement | LLM Stats |
| DeepSeek-V4-Flash-0423 | 0.864 | Not recorded | Not recorded | 2026-04-23 · model_announcement | LLM Stats |
| Solar Pro 4 | 0.863 | Not recorded | Not recorded | 2026-08-06 · model_announcement | LLM Stats |
| DeepSeek-V4-Flash-Max | 0.862 | Not recorded | Not recorded | 2026-04-23 · model_announcement | LLM Stats |
| Qwen3.6-27B | 0.862 | Not recorded | Not recorded | 2026-04-21 · model_announcement | LLM Stats |
| Qwen3.5-27B | 0.861 | Not recorded | Not recorded | 2026-02-24 · model_announcement | LLM Stats |
| Qwen3.5-35B-A3B | 0.853 | Not recorded | Not recorded | 2026-02-24 · model_announcement | LLM Stats |
| Qwen3.6-35B-A3B | 0.852 | Not recorded | Not recorded | 2026-04-16 · model_announcement | LLM Stats |
| Gemma 4 31B | 0.852 | Not recorded | Not recorded | 2026-04-02 · model_announcement | LLM Stats |
| MAI-Thinking-1 | 0.85 | Not recorded | Not recorded | 2026-06-02 · model_announcement | LLM Stats |
| DeepSeek-V3.2 (Thinking) | 0.85 | Not recorded | Not recorded | 2025-12-01 · model_announcement | LLM Stats |
| DeepSeek-V3.2 | 0.85 | Not recorded | Not recorded | 2025-12-01 · model_announcement | LLM Stats |
| DeepSeek-V3.2-Exp | 0.85 | Not recorded | Not recorded | 2025-09-29 · model_announcement | LLM Stats |
| DeepSeek-R1-0528 | 0.85 | Not recorded | Not recorded | 2025-05-28 · model_announcement | LLM Stats |
| MiMo-V2-Flash | 0.849 | Not recorded | Not recorded | 2025-12-16 · model_announcement | LLM Stats |
| Kimi K2-Thinking-0905 | 0.846 | Not recorded | Not recorded | 2025-09-05 · model_announcement | LLM Stats |
| GLM-4.5 | 0.846 | Not recorded | Not recorded | 2025-07-28 · model_announcement | LLM Stats |
| Qwen3-235B-A22B-Thinking-2507 | 0.844 | Not recorded | Not recorded | 2025-07-25 · model_announcement | LLM Stats |
| GLM-4.7 | 0.843 | Not recorded | Not recorded | 2025-12-22 · model_announcement | LLM Stats |
| K-EXAONE-236B-A23B | 0.838 | Not recorded | Not recorded | 2025-12-31 · model_announcement | LLM Stats |
| Qwen3 VL 235B A22B Thinking | 0.838 | Not recorded | Not recorded | 2025-09-22 · model_announcement | LLM Stats |
| Nemotron 3 Super (120B A12B) | 0.8373 | Not recorded | Not recorded | 2026-03-11 · model_announcement | LLM Stats |
| DeepSeek-V3.1 | 0.837 | Not recorded | Not recorded | 2025-01-10 · model_announcement | LLM Stats |
| Qwen3-235B-A22B-Instruct-2507 | 0.83 | Not recorded | Not recorded | 2025-07-22 · model_announcement | LLM Stats |
| Qwen3-Next-80B-A3B-Thinking | 0.827 | Not recorded | Not recorded | 2025-09-10 · model_announcement | LLM Stats |
| LongCat-Flash-Chat | 0.8268 | Not recorded | Not recorded | 2025-08-29 · model_announcement | LLM Stats |
| Gemma 4 26B-A4B | 0.826 | Not recorded | Not recorded | 2026-04-02 · model_announcement | LLM Stats |
| LongCat-Flash-Thinking | 0.826 | Not recorded | Not recorded | 2025-09-22 · model_announcement | LLM Stats |
| Qwen3.5-9B | 0.825 | Not recorded | Not recorded | 2026-03-02 · model_announcement | LLM Stats |
| Kimi K2 0905 | 0.825 | Not recorded | Not recorded | 2025-09-05 · model_announcement | LLM Stats |
| Qwen3 VL 32B Thinking | 0.821 | Not recorded | Not recorded | 2025-09-22 · model_announcement | LLM Stats |
| MiniMax M2 | 0.82 | Not recorded | Not recorded | 2025-10-27 · model_announcement | LLM Stats |
| Nemotron 3.5 Lightning (30B A3B) | 0.8194 | Not recorded | Not recorded | 2026-08-11 · model_announcement | LLM Stats |
| Qwen3 VL 235B A22B Instruct | 0.818 | Not recorded | Not recorded | 2025-09-22 · model_announcement | LLM Stats |
| Sarvam-105B | 0.817 | Not recorded | Not recorded | 2026-03-06 · model_announcement | LLM Stats |
| Nova 2 Pro | 0.816 | Not recorded | Not recorded | 2025-12-02 · model_announcement | LLM Stats |
| GLM-4.5-Air | 0.814 | Not recorded | Not recorded | 2025-07-28 · model_announcement | LLM Stats |
| DeepSeek-V3 0324 | 0.812 | Not recorded | Not recorded | 2025-03-25 · model_announcement | LLM Stats |
| Kimi K2-Instruct-0905 | 0.811 | Not recorded | Not recorded | 2025-09-05 · model_announcement | LLM Stats |
| Kimi K2 Instruct | 0.811 | Not recorded | Not recorded | 2025-07-11 · model_announcement | LLM Stats |
| MiniMax M1 80K | 0.811 | Not recorded | Not recorded | 2025-06-16 · model_announcement | LLM Stats |
| Nova 2 Lite | 0.809 | Not recorded | Not recorded | 2025-12-02 · model_announcement | LLM Stats |
| Nova 2 Omni | 0.807 | Not recorded | Not recorded | 2025-12-02 · model_announcement | LLM Stats |
| GPT OSS 120B High | 0.807 | Not recorded | Not recorded | 2025-08-05 · model_announcement | LLM Stats |
| Qwen3-Next-80B-A3B-Instruct | 0.806 | Not recorded | Not recorded | 2025-09-10 · model_announcement | LLM Stats |
| MiniMax M1 40K | 0.806 | Not recorded | Not recorded | 2025-06-16 · model_announcement | LLM Stats |
| Qwen3 VL 30B A3B Thinking | 0.805 | Not recorded | Not recorded | 2025-09-22 · model_announcement | LLM Stats |
| Llama 4 Maverick | 0.805 | Not recorded | Not recorded | 2025-04-05 · model_announcement | LLM Stats |
| Sarvam-30B | 0.8 | Not recorded | Not recorded | 2026-03-06 · model_announcement | LLM Stats |
| Qwen3.5-4B | 0.791 | Not recorded | Not recorded | 2026-03-02 · model_announcement | LLM Stats |
| Qwen3 VL 32B Instruct | 0.786 | Not recorded | Not recorded | 2025-09-22 · model_announcement | LLM Stats |
| Nemotron 3 Nano (30B A3B) | 0.783 | Not recorded | Not recorded | 2025-12-15 · model_announcement | LLM Stats |
| LongCat-Flash-Lite | 0.7829 | Not recorded | Not recorded | 2026-02-05 · model_announcement | LLM Stats |
| Mistral Small 4 | 0.78 | Not recorded | Not recorded | 2026-03-16 · model_announcement | LLM Stats |
| Qwen3 VL 30B A3B Instruct | 0.778 | Not recorded | Not recorded | 2025-09-22 · model_announcement | LLM Stats |
| DiffusionGemma 26B-A4B | 0.776 | Not recorded | Not recorded | 2026-06-10 · model_announcement | LLM Stats |
| Claude 3.5 Sonnet | 0.776 | Not recorded | Not recorded | 2024-10-22 · model_announcement | LLM Stats |
| Qwen3 VL 8B Thinking | 0.773 | Not recorded | Not recorded | 2025-09-22 · model_announcement | LLM Stats |
| Gemma 4 12B | 0.772 | Not recorded | Not recorded | 2026-05-23 · model_announcement | LLM Stats |
| Gemini 2.0 Flash | 0.764 | Not recorded | Not recorded | 2024-12-01 · model_announcement | LLM Stats |
| Claude 3.5 Sonnet | 0.761 | Not recorded | Not recorded | 2024-06-21 · model_announcement | LLM Stats |
| Phi 4 Reasoning Plus | 0.76 | Not recorded | Not recorded | 2025-04-30 · model_announcement | LLM Stats |
| DeepSeek-V3 | 0.759 | Not recorded | Not recorded | 2024-12-25 · model_announcement | LLM Stats |
| Gemini 1.5 Pro | 0.758 | Not recorded | Not recorded | 2024-05-01 · model_announcement | LLM Stats |
| Grok-2 | 0.755 | Not recorded | Not recorded | 2024-08-13 · model_announcement | LLM Stats |
| GPT-4o | 0.747 | Not recorded | Not recorded | 2024-08-06 · model_announcement | LLM Stats |
| Phi 4 Reasoning | 0.743 | Not recorded | Not recorded | 2025-04-30 · model_announcement | LLM Stats |
| Llama 4 Scout | 0.743 | Not recorded | Not recorded | 2025-04-05 · model_announcement | LLM Stats |
| Qwen3 VL 4B Thinking | 0.736 | Not recorded | Not recorded | 2025-09-22 · model_announcement | LLM Stats |
| Llama 3.1 405B Instruct | 0.733 | Not recorded | Not recorded | 2024-07-23 · model_announcement | LLM Stats |
| GPT-4o | 0.726 | Not recorded | Not recorded | 2024-05-13 · model_announcement | LLM Stats |
| Grok-2 mini | 0.72 | Not recorded | Not recorded | 2024-08-13 · model_announcement | LLM Stats |
| Qwen3 VL 8B Instruct | 0.716 | Not recorded | Not recorded | 2025-09-22 · model_announcement | LLM Stats |
| Gemini 2.0 Flash-Lite | 0.716 | Not recorded | Not recorded | 2025-02-05 · model_announcement | LLM Stats |
| Qwen2.5 72B Instruct | 0.711 | Not recorded | Not recorded | 2024-09-19 · model_announcement | LLM Stats |
| Phi 4 | 0.704 | Not recorded | Not recorded | 2024-12-12 · model_announcement | LLM Stats |
| Gemma 4 E4B | 0.694 | Not recorded | Not recorded | 2026-04-02 · model_announcement | LLM Stats |
| Kimi K2 Base | 0.692 | Not recorded | Not recorded | 2025-07-11 · model_announcement | LLM Stats |
| Mistral Small 3.2 24B Instruct | 0.6906 | Not recorded | Not recorded | 2025-06-20 · model_announcement | LLM Stats |
| Qwen2.5 32B Instruct | 0.69 | Not recorded | Not recorded | 2024-09-19 · model_announcement | LLM Stats |
| Llama 3.3 70B Instruct | 0.689 | Not recorded | Not recorded | 2024-12-06 · model_announcement | LLM Stats |
| Qwen2.5 VL 32B Instruct | 0.688 | Not recorded | Not recorded | 2025-02-28 · model_announcement | LLM Stats |
| MiMo-V2.5-Pro | 0.685 | Not recorded | Not recorded | 2026-04-27 · model_announcement | LLM Stats |
| Claude 3 Opus | 0.685 | Not recorded | Not recorded | 2024-02-29 · model_announcement | LLM Stats |
| Qwen3 235B A22B | 0.6818 | Not recorded | Not recorded | 2025-04-29 · model_announcement | LLM Stats |
| Gemma 3 27B | 0.675 | Not recorded | Not recorded | 2025-03-12 · model_announcement | LLM Stats |
| Gemini 1.5 Flash | 0.673 | Not recorded | Not recorded | 2024-05-01 · model_announcement | LLM Stats |
| Qwen3 VL 4B Instruct | 0.671 | Not recorded | Not recorded | 2025-09-22 · model_announcement | LLM Stats |
| MiniCPM-SALA | 0.6704 | Not recorded | Not recorded | 2026-02-11 · model_announcement | LLM Stats |
| Mistral Small 3.1 24B Instruct | 0.6676 | Not recorded | Not recorded | 2025-03-17 · model_announcement | LLM Stats |
| Qwen3.5-2B | 0.665 | Not recorded | Not recorded | 2026-03-02 · model_announcement | LLM Stats |
| Llama 3.1 70B Instruct | 0.664 | Not recorded | Not recorded | 2024-07-23 · model_announcement | LLM Stats |
| Mistral Small 3 24B Instruct | 0.663 | Not recorded | Not recorded | 2025-01-30 · model_announcement | LLM Stats |
| Claude 3.5 Haiku | 0.65 | Not recorded | Not recorded | 2024-10-22 · model_announcement | LLM Stats |
| Qwen2 72B Instruct | 0.644 | Not recorded | Not recorded | 2024-07-23 · model_announcement | LLM Stats |
| Qwen2.5 14B Instruct | 0.637 | Not recorded | Not recorded | 2024-09-19 · model_announcement | LLM Stats |
| Gemma 3 12B | 0.606 | Not recorded | Not recorded | 2025-03-12 · model_announcement | LLM Stats |
| Gemma 4 E2B | 0.6 | Not recorded | Not recorded | 2026-04-02 · model_announcement | LLM Stats |
| Gemini 1.5 Flash 8B | 0.587 | Not recorded | Not recorded | 2024-03-15 · model_announcement | LLM Stats |
| Claude 3 Sonnet | 0.568 | Not recorded | Not recorded | 2024-02-29 · model_announcement | LLM Stats |
| Qwen2.5 7B Instruct | 0.563 | Not recorded | Not recorded | 2024-09-19 · model_announcement | LLM Stats |
| Mistral Small 3.1 24B Base | 0.5603 | Not recorded | Not recorded | 2025-03-17 · model_announcement | LLM Stats |
| Mistral Small 3 24B Base | 0.5437 | Not recorded | Not recorded | 2025-01-30 · model_announcement | LLM Stats |
| Jamba 1.5 Large | 0.535 | Not recorded | Not recorded | 2024-08-22 · model_announcement | LLM Stats |
| Phi 4 Mini | 0.528 | Not recorded | Not recorded | 2025-02-01 · model_announcement | LLM Stats |
| Grok-1.5 | 0.51 | Not recorded | Not recorded | 2024-03-28 · model_announcement | LLM Stats |
| Gemma 3n E4B Instructed | 0.506 | Not recorded | Not recorded | 2025-06-26 · model_announcement | LLM Stats |
| Gemma 3n E4B Instructed LiteRT Preview | 0.506 | Not recorded | Not recorded | 2025-05-20 · model_announcement | LLM Stats |
| Qwen2.5-Coder 32B Instruct | 0.504 | Not recorded | Not recorded | 2024-09-19 · model_announcement | LLM Stats |
| Llama 3.1 8B Instruct | 0.483 | Not recorded | Not recorded | 2024-07-23 · model_announcement | LLM Stats |
| Phi-3.5-mini-instruct | 0.474 | Not recorded | Not recorded | 2024-08-23 · model_announcement | LLM Stats |
| Hermes 3 70B | 0.4724 | Not recorded | Not recorded | 2024-08-15 · model_announcement | LLM Stats |
| Qwen2.5-Omni-7B | 0.47 | Not recorded | Not recorded | 2025-03-27 · model_announcement | LLM Stats |
| Phi-3.5-MoE-instruct | 0.453 | Not recorded | Not recorded | 2024-08-23 · model_announcement | LLM Stats |
| Qwen2 7B Instruct | 0.441 | Not recorded | Not recorded | 2024-07-23 · model_announcement | LLM Stats |
| Gemma 3 4B | 0.436 | Not recorded | Not recorded | 2025-03-12 · model_announcement | LLM Stats |
| Jamba 1.5 Mini | 0.425 | Not recorded | Not recorded | 2024-08-22 · model_announcement | LLM Stats |
| Qwen3.5-0.8B | 0.423 | Not recorded | Not recorded | 2026-03-02 · model_announcement | LLM Stats |
| Gemma 3n E2B Instructed | 0.405 | Not recorded | Not recorded | 2025-06-26 · model_announcement | LLM Stats |
| Gemma 3n E2B Instructed LiteRT (Preview) | 0.405 | Not recorded | Not recorded | 2025-05-20 · model_announcement | LLM Stats |
| Qwen2.5-Coder 7B Instruct | 0.401 | Not recorded | Not recorded | 2024-09-19 · model_announcement | LLM Stats |
| North Micro Vision Instruct | 0.307 | Not recorded | Not recorded | 2026-08-12 · model_announcement | LLM Stats |
| ERNIE 4.5 | 0.16 | Not recorded | Not recorded | 2025-06-25 · model_announcement | LLM Stats |
| Gemma 3 1B | 0.147 | Not recorded | Not recorded | 2025-03-12 · model_announcement | LLM Stats |