LLM Stats

MMLU-Pro

MMLU-Pro combines academic knowledge and reasoning across many subjects. It expands answer choices and revises the question pool, so its accuracy should not be read as the same test as MMLU.

Catalog source

AI Benchmark Radar / Equality Machine · Reviewed 2026-10-25

Original MMLU-Pro and its official evaluation resources; not original MMLU, MMLU-ProX or multilingual derivatives.

Harder multiple-choice questions

The benchmark increases the option set from MMLU's four to ten and emphasizes reasoning-intensive questions. It filters trivial or noisy items and adds material from exams and textbooks, changing both task composition and guessing difficulty.

MMLU-Pro paper · Construction, prompts and limitations

Read subjects as well as the total

The official release contains more than 12,000 questions across 14 domains, including mathematics, law, health and engineering. The dataset includes a category field, so a broad average can be supplemented by the subjects relevant to your application.

TIGER AI Lab · MMLU-Pro evaluation and answer extraction · TIGER Lab · Official MMLU-Pro dataset and schema

From generated text to an answer option

Evaluation extracts an answer option from generated text and checks it against the reference. The repository provides accuracy calculation and alternative extraction procedures. Preserve the extractor and prompting setup when reproducing a reported number.

TIGER AI Lab · MMLU-Pro evaluation and answer extraction

Reasoning prompts remain part of the protocol

The paper studies both direct answering and chain-of-thought prompting, and tests variation across prompt styles. Greater robustness does not mean prompts are irrelevant. Compare models under the same examples and answer format before interpreting a difference.

MMLU-Pro paper · Construction, prompts and limitations

A broad exam is still a bounded task

Choosing a correct option is narrower than producing a useful professional deliverable. Category scores and example errors reveal more than the overall percentage alone. Keep the original test distinct from later variants with different languages or question pools.

TIGER Lab · Official MMLU-Pro dataset and schema · MMLU-Pro paper · Construction, prompts and limitations

Data source and credit: LLM Stats

Evidence available for this benchmark

This source record contains 134 numeric observations. The source distinguishes 134 model identities or configurations. A model may appear more than once; check the version, tools and run budget before comparing. Read results and row-level evidence →

Metric direction: higher is better

Start with these resources

Paper · task design and experiments · arxiv.org
Code · implementation and instructions · github.com
Dataset · files and documentation · huggingface.co
Catalog source · api.zeroeval.com

Cited documents and dates (1)

MMLU-Pro

Catalog collected on 2026-08-17; resources and fields reflect that saved source.

Recorded attention

Observed on: 2026-10-12

GitHub stars: 430
Hugging Face paper upvotes: 56

tiger-ai-lab/mmlu-pro · View source snapshot

Dated attention observations, not live counts or benchmark scores.

Evaluation context and evidence

Source: LLM Stats
Publisher: University of Waterloo
Benchmark release: 2024-06-03
Collected: 2026-08-17
Modality: text

These are source-reported observations, not independent reruns by this site. Check the task version, evaluation protocol, unit and reporting date before comparing models. Results from different setups are not a universal ranking.

Reported results 134

Shown in source observation order, not ranked. Dates retain the precision provided by the source.

Model	Value	Unit	Protocol	Reported date and basis	Source
Sakana Namazu	0.9033	Not recorded	Not recorded	2026-08-03 · model_announcement	LLM Stats
Qwen3.7 Max	0.896	Not recorded	Not recorded	2026-05-19 · model_announcement	LLM Stats
Qwen3.7-Plus	0.885	Not recorded	Not recorded	2026-05-31 · model_announcement	LLM Stats
Qwen3.6 Plus	0.885	Not recorded	Not recorded	2026-04-02 · model_announcement	LLM Stats
MiniMax M2.1	0.88	Not recorded	Not recorded	2025-12-23 · model_announcement	LLM Stats
Qwen3.5-397B-A17B	0.878	Not recorded	Not recorded	2026-02-16 · model_announcement	LLM Stats
DeepSeek-V4-Pro-Max	0.875	Not recorded	Not recorded	2026-04-23 · model_announcement	LLM Stats
Kimi K2.5	0.871	Not recorded	Not recorded	2026-01-27 · model_announcement	LLM Stats
ERNIE 5.0	0.87	Not recorded	Not recorded	2026-01-22 · model_announcement	LLM Stats
Nemotron 3 Ultra (550B A55B)	0.868	Not recorded	Not recorded	2026-06-04 · model_announcement	LLM Stats
Qwen3.5-122B-A10B	0.867	Not recorded	Not recorded	2026-02-24 · model_announcement	LLM Stats
DeepSeek-V4-Flash-0423	0.864	Not recorded	Not recorded	2026-04-23 · model_announcement	LLM Stats
Solar Pro 4	0.863	Not recorded	Not recorded	2026-08-06 · model_announcement	LLM Stats
DeepSeek-V4-Flash-Max	0.862	Not recorded	Not recorded	2026-04-23 · model_announcement	LLM Stats
Qwen3.6-27B	0.862	Not recorded	Not recorded	2026-04-21 · model_announcement	LLM Stats
Qwen3.5-27B	0.861	Not recorded	Not recorded	2026-02-24 · model_announcement	LLM Stats
Qwen3.5-35B-A3B	0.853	Not recorded	Not recorded	2026-02-24 · model_announcement	LLM Stats
Qwen3.6-35B-A3B	0.852	Not recorded	Not recorded	2026-04-16 · model_announcement	LLM Stats
Gemma 4 31B	0.852	Not recorded	Not recorded	2026-04-02 · model_announcement	LLM Stats
MAI-Thinking-1	0.85	Not recorded	Not recorded	2026-06-02 · model_announcement	LLM Stats
DeepSeek-V3.2 (Thinking)	0.85	Not recorded	Not recorded	2025-12-01 · model_announcement	LLM Stats
DeepSeek-V3.2	0.85	Not recorded	Not recorded	2025-12-01 · model_announcement	LLM Stats
DeepSeek-V3.2-Exp	0.85	Not recorded	Not recorded	2025-09-29 · model_announcement	LLM Stats
DeepSeek-R1-0528	0.85	Not recorded	Not recorded	2025-05-28 · model_announcement	LLM Stats
MiMo-V2-Flash	0.849	Not recorded	Not recorded	2025-12-16 · model_announcement	LLM Stats
Kimi K2-Thinking-0905	0.846	Not recorded	Not recorded	2025-09-05 · model_announcement	LLM Stats
GLM-4.5	0.846	Not recorded	Not recorded	2025-07-28 · model_announcement	LLM Stats
Qwen3-235B-A22B-Thinking-2507	0.844	Not recorded	Not recorded	2025-07-25 · model_announcement	LLM Stats
GLM-4.7	0.843	Not recorded	Not recorded	2025-12-22 · model_announcement	LLM Stats
K-EXAONE-236B-A23B	0.838	Not recorded	Not recorded	2025-12-31 · model_announcement	LLM Stats
Qwen3 VL 235B A22B Thinking	0.838	Not recorded	Not recorded	2025-09-22 · model_announcement	LLM Stats
Nemotron 3 Super (120B A12B)	0.8373	Not recorded	Not recorded	2026-03-11 · model_announcement	LLM Stats
DeepSeek-V3.1	0.837	Not recorded	Not recorded	2025-01-10 · model_announcement	LLM Stats
Qwen3-235B-A22B-Instruct-2507	0.83	Not recorded	Not recorded	2025-07-22 · model_announcement	LLM Stats
Qwen3-Next-80B-A3B-Thinking	0.827	Not recorded	Not recorded	2025-09-10 · model_announcement	LLM Stats
LongCat-Flash-Chat	0.8268	Not recorded	Not recorded	2025-08-29 · model_announcement	LLM Stats
Gemma 4 26B-A4B	0.826	Not recorded	Not recorded	2026-04-02 · model_announcement	LLM Stats
LongCat-Flash-Thinking	0.826	Not recorded	Not recorded	2025-09-22 · model_announcement	LLM Stats
Qwen3.5-9B	0.825	Not recorded	Not recorded	2026-03-02 · model_announcement	LLM Stats
Kimi K2 0905	0.825	Not recorded	Not recorded	2025-09-05 · model_announcement	LLM Stats
Qwen3 VL 32B Thinking	0.821	Not recorded	Not recorded	2025-09-22 · model_announcement	LLM Stats
MiniMax M2	0.82	Not recorded	Not recorded	2025-10-27 · model_announcement	LLM Stats
Nemotron 3.5 Lightning (30B A3B)	0.8194	Not recorded	Not recorded	2026-08-11 · model_announcement	LLM Stats
Qwen3 VL 235B A22B Instruct	0.818	Not recorded	Not recorded	2025-09-22 · model_announcement	LLM Stats
Sarvam-105B	0.817	Not recorded	Not recorded	2026-03-06 · model_announcement	LLM Stats
Nova 2 Pro	0.816	Not recorded	Not recorded	2025-12-02 · model_announcement	LLM Stats
GLM-4.5-Air	0.814	Not recorded	Not recorded	2025-07-28 · model_announcement	LLM Stats
DeepSeek-V3 0324	0.812	Not recorded	Not recorded	2025-03-25 · model_announcement	LLM Stats
Kimi K2-Instruct-0905	0.811	Not recorded	Not recorded	2025-09-05 · model_announcement	LLM Stats
Kimi K2 Instruct	0.811	Not recorded	Not recorded	2025-07-11 · model_announcement	LLM Stats
MiniMax M1 80K	0.811	Not recorded	Not recorded	2025-06-16 · model_announcement	LLM Stats
Nova 2 Lite	0.809	Not recorded	Not recorded	2025-12-02 · model_announcement	LLM Stats
Nova 2 Omni	0.807	Not recorded	Not recorded	2025-12-02 · model_announcement	LLM Stats
GPT OSS 120B High	0.807	Not recorded	Not recorded	2025-08-05 · model_announcement	LLM Stats
Qwen3-Next-80B-A3B-Instruct	0.806	Not recorded	Not recorded	2025-09-10 · model_announcement	LLM Stats
MiniMax M1 40K	0.806	Not recorded	Not recorded	2025-06-16 · model_announcement	LLM Stats
Qwen3 VL 30B A3B Thinking	0.805	Not recorded	Not recorded	2025-09-22 · model_announcement	LLM Stats
Llama 4 Maverick	0.805	Not recorded	Not recorded	2025-04-05 · model_announcement	LLM Stats
Sarvam-30B	0.8	Not recorded	Not recorded	2026-03-06 · model_announcement	LLM Stats
Qwen3.5-4B	0.791	Not recorded	Not recorded	2026-03-02 · model_announcement	LLM Stats
Qwen3 VL 32B Instruct	0.786	Not recorded	Not recorded	2025-09-22 · model_announcement	LLM Stats
Nemotron 3 Nano (30B A3B)	0.783	Not recorded	Not recorded	2025-12-15 · model_announcement	LLM Stats
LongCat-Flash-Lite	0.7829	Not recorded	Not recorded	2026-02-05 · model_announcement	LLM Stats
Mistral Small 4	0.78	Not recorded	Not recorded	2026-03-16 · model_announcement	LLM Stats
Qwen3 VL 30B A3B Instruct	0.778	Not recorded	Not recorded	2025-09-22 · model_announcement	LLM Stats
DiffusionGemma 26B-A4B	0.776	Not recorded	Not recorded	2026-06-10 · model_announcement	LLM Stats
Claude 3.5 Sonnet	0.776	Not recorded	Not recorded	2024-10-22 · model_announcement	LLM Stats
Qwen3 VL 8B Thinking	0.773	Not recorded	Not recorded	2025-09-22 · model_announcement	LLM Stats
Gemma 4 12B	0.772	Not recorded	Not recorded	2026-05-23 · model_announcement	LLM Stats
Gemini 2.0 Flash	0.764	Not recorded	Not recorded	2024-12-01 · model_announcement	LLM Stats
Claude 3.5 Sonnet	0.761	Not recorded	Not recorded	2024-06-21 · model_announcement	LLM Stats
Phi 4 Reasoning Plus	0.76	Not recorded	Not recorded	2025-04-30 · model_announcement	LLM Stats
DeepSeek-V3	0.759	Not recorded	Not recorded	2024-12-25 · model_announcement	LLM Stats
Gemini 1.5 Pro	0.758	Not recorded	Not recorded	2024-05-01 · model_announcement	LLM Stats
Grok-2	0.755	Not recorded	Not recorded	2024-08-13 · model_announcement	LLM Stats
GPT-4o	0.747	Not recorded	Not recorded	2024-08-06 · model_announcement	LLM Stats
Phi 4 Reasoning	0.743	Not recorded	Not recorded	2025-04-30 · model_announcement	LLM Stats
Llama 4 Scout	0.743	Not recorded	Not recorded	2025-04-05 · model_announcement	LLM Stats
Qwen3 VL 4B Thinking	0.736	Not recorded	Not recorded	2025-09-22 · model_announcement	LLM Stats
Llama 3.1 405B Instruct	0.733	Not recorded	Not recorded	2024-07-23 · model_announcement	LLM Stats
GPT-4o	0.726	Not recorded	Not recorded	2024-05-13 · model_announcement	LLM Stats
Grok-2 mini	0.72	Not recorded	Not recorded	2024-08-13 · model_announcement	LLM Stats
Qwen3 VL 8B Instruct	0.716	Not recorded	Not recorded	2025-09-22 · model_announcement	LLM Stats
Gemini 2.0 Flash-Lite	0.716	Not recorded	Not recorded	2025-02-05 · model_announcement	LLM Stats
Qwen2.5 72B Instruct	0.711	Not recorded	Not recorded	2024-09-19 · model_announcement	LLM Stats
Phi 4	0.704	Not recorded	Not recorded	2024-12-12 · model_announcement	LLM Stats
Gemma 4 E4B	0.694	Not recorded	Not recorded	2026-04-02 · model_announcement	LLM Stats
Kimi K2 Base	0.692	Not recorded	Not recorded	2025-07-11 · model_announcement	LLM Stats
Mistral Small 3.2 24B Instruct	0.6906	Not recorded	Not recorded	2025-06-20 · model_announcement	LLM Stats
Qwen2.5 32B Instruct	0.69	Not recorded	Not recorded	2024-09-19 · model_announcement	LLM Stats
Llama 3.3 70B Instruct	0.689	Not recorded	Not recorded	2024-12-06 · model_announcement	LLM Stats
Qwen2.5 VL 32B Instruct	0.688	Not recorded	Not recorded	2025-02-28 · model_announcement	LLM Stats
MiMo-V2.5-Pro	0.685	Not recorded	Not recorded	2026-04-27 · model_announcement	LLM Stats
Claude 3 Opus	0.685	Not recorded	Not recorded	2024-02-29 · model_announcement	LLM Stats
Qwen3 235B A22B	0.6818	Not recorded	Not recorded	2025-04-29 · model_announcement	LLM Stats
Gemma 3 27B	0.675	Not recorded	Not recorded	2025-03-12 · model_announcement	LLM Stats
Gemini 1.5 Flash	0.673	Not recorded	Not recorded	2024-05-01 · model_announcement	LLM Stats
Qwen3 VL 4B Instruct	0.671	Not recorded	Not recorded	2025-09-22 · model_announcement	LLM Stats
MiniCPM-SALA	0.6704	Not recorded	Not recorded	2026-02-11 · model_announcement	LLM Stats
Mistral Small 3.1 24B Instruct	0.6676	Not recorded	Not recorded	2025-03-17 · model_announcement	LLM Stats
Qwen3.5-2B	0.665	Not recorded	Not recorded	2026-03-02 · model_announcement	LLM Stats
Llama 3.1 70B Instruct	0.664	Not recorded	Not recorded	2024-07-23 · model_announcement	LLM Stats
Mistral Small 3 24B Instruct	0.663	Not recorded	Not recorded	2025-01-30 · model_announcement	LLM Stats
Claude 3.5 Haiku	0.65	Not recorded	Not recorded	2024-10-22 · model_announcement	LLM Stats
Qwen2 72B Instruct	0.644	Not recorded	Not recorded	2024-07-23 · model_announcement	LLM Stats
Qwen2.5 14B Instruct	0.637	Not recorded	Not recorded	2024-09-19 · model_announcement	LLM Stats
Gemma 3 12B	0.606	Not recorded	Not recorded	2025-03-12 · model_announcement	LLM Stats
Gemma 4 E2B	0.6	Not recorded	Not recorded	2026-04-02 · model_announcement	LLM Stats
Gemini 1.5 Flash 8B	0.587	Not recorded	Not recorded	2024-03-15 · model_announcement	LLM Stats
Claude 3 Sonnet	0.568	Not recorded	Not recorded	2024-02-29 · model_announcement	LLM Stats
Qwen2.5 7B Instruct	0.563	Not recorded	Not recorded	2024-09-19 · model_announcement	LLM Stats
Mistral Small 3.1 24B Base	0.5603	Not recorded	Not recorded	2025-03-17 · model_announcement	LLM Stats
Mistral Small 3 24B Base	0.5437	Not recorded	Not recorded	2025-01-30 · model_announcement	LLM Stats
Jamba 1.5 Large	0.535	Not recorded	Not recorded	2024-08-22 · model_announcement	LLM Stats
Phi 4 Mini	0.528	Not recorded	Not recorded	2025-02-01 · model_announcement	LLM Stats
Grok-1.5	0.51	Not recorded	Not recorded	2024-03-28 · model_announcement	LLM Stats
Gemma 3n E4B Instructed	0.506	Not recorded	Not recorded	2025-06-26 · model_announcement	LLM Stats
Gemma 3n E4B Instructed LiteRT Preview	0.506	Not recorded	Not recorded	2025-05-20 · model_announcement	LLM Stats
Qwen2.5-Coder 32B Instruct	0.504	Not recorded	Not recorded	2024-09-19 · model_announcement	LLM Stats
Llama 3.1 8B Instruct	0.483	Not recorded	Not recorded	2024-07-23 · model_announcement	LLM Stats
Phi-3.5-mini-instruct	0.474	Not recorded	Not recorded	2024-08-23 · model_announcement	LLM Stats
Hermes 3 70B	0.4724	Not recorded	Not recorded	2024-08-15 · model_announcement	LLM Stats
Qwen2.5-Omni-7B	0.47	Not recorded	Not recorded	2025-03-27 · model_announcement	LLM Stats
Phi-3.5-MoE-instruct	0.453	Not recorded	Not recorded	2024-08-23 · model_announcement	LLM Stats
Qwen2 7B Instruct	0.441	Not recorded	Not recorded	2024-07-23 · model_announcement	LLM Stats
Gemma 3 4B	0.436	Not recorded	Not recorded	2025-03-12 · model_announcement	LLM Stats
Jamba 1.5 Mini	0.425	Not recorded	Not recorded	2024-08-22 · model_announcement	LLM Stats
Qwen3.5-0.8B	0.423	Not recorded	Not recorded	2026-03-02 · model_announcement	LLM Stats
Gemma 3n E2B Instructed	0.405	Not recorded	Not recorded	2025-06-26 · model_announcement	LLM Stats
Gemma 3n E2B Instructed LiteRT (Preview)	0.405	Not recorded	Not recorded	2025-05-20 · model_announcement	LLM Stats
Qwen2.5-Coder 7B Instruct	0.401	Not recorded	Not recorded	2024-09-19 · model_announcement	LLM Stats
North Micro Vision Instruct	0.307	Not recorded	Not recorded	2026-08-12 · model_announcement	LLM Stats
ERNIE 4.5	0.16	Not recorded	Not recorded	2025-06-25 · model_announcement	LLM Stats
Gemma 3 1B	0.147	Not recorded	Not recorded	2025-03-12 · model_announcement	LLM Stats

Related records in other sources

MMLU-Pro

Harder multiple-choice questions

Read subjects as well as the total

From generated text to an answer option

Reasoning prompts remain part of the protocol

A broad exam is still a bounded task

Evidence available for this benchmark

Start with these resources

Recorded attention

Evaluation context and evidence

Reported results 134

Explore related benchmarks

Put this benchmark in context

Six Navers Lab / Einsia benchmarks and the work they test

AI benchmarks and leaderboards are different things

How to compare AI benchmark scores without mixing protocols

Four new agent benchmarks: adaptation, safety and video

Continue exploring

Related records in other sources