LLM Stats

GPQA

GPQA tests difficult specialist science questions with multiple-choice answers. Diamond is a selected subset, so identifying the exact split is more useful than comparing the benchmark name alone.

Catalog source

AI Benchmark Radar / Equality Machine · Reviewed 2026-10-25

The original GPQA family, including the main, Extended and Diamond sets. A shared paper does not establish that two reported scores use the same split.

Specialist science questions

Experts wrote and validated questions in biology, chemistry and physics. Each question has a correct option and distractors. The task tests choosing the answer to a specialized problem, rather than performing a laboratory experiment or producing a research contribution.

GPQA paper · Dataset splits and limitations · GPQA · Author dataset card and exposure notice

Main, Extended and Diamond

The paper distinguishes 546 Extended questions, 448 main questions and 198 Diamond questions. Diamond applies stricter expert-agreement and non-expert-difficulty criteria. A label such as full GPQA is insufficient: check the dataset configuration or reported question count.

GPQA paper · Dataset splits and limitations

Accuracy needs an evaluation setup

Accuracy measures the fraction of selected answers that are correct. The released baselines include different prompting and open-book configurations. Keep answer extraction, examples, search access and repeat policy visible when comparing scores from separate reports.

GPQA · Official baselines and prompts

What Google-proof means here

The name reflects difficulty for skilled non-experts even when they could search the web during construction. It is not a guarantee against training contamination, nor evidence that retrieval never helps a later system.

GPQA · Author dataset card and exposure notice

Small datasets and exposure

The authors flag limited statistical power for small accuracy differences. The dataset page also asks users not to republish questions to reduce training leakage. Compare uncertainty and failure cases, and avoid interpreting a small lead as general scientific competence.

GPQA paper · Dataset splits and limitations · GPQA · Author dataset card and exposure notice

Data source and credit: LLM Stats

Evidence available for this benchmark

This source record contains 239 numeric observations. The source distinguishes 239 model identities or configurations. A model may appear more than once; check the version, tools and run budget before comparing. Read results and row-level evidence →

Metric direction: higher is better

Start with these resources

Paper · task design and experiments · arxiv.org
Code · implementation and instructions · github.com
Dataset · files and documentation · huggingface.co
Catalog source · api.zeroeval.com

Cited documents and dates (1)

GPQA

Catalog collected on 2026-08-17; resources and fields reflect that saved source.

Recorded attention

Observed on: 2026-10-12

GitHub stars: 540
Hugging Face paper upvotes: 38

idavidrein/gpqa · View source snapshot

Dated attention observations, not live counts or benchmark scores.

Evaluation context and evidence

Source: LLM Stats
Publisher: Anthropic
Benchmark release: 2023-11-20
Collected: 2026-08-17
Modality: text

These are source-reported observations, not independent reruns by this site. Check the task version, evaluation protocol, unit and reporting date before comparing models. Results from different setups are not a universal ranking.

Reported results 239

Shown in source observation order, not ranked. Dates retain the precision provided by the source.

Model	Value	Unit	Protocol	Reported date and basis	Source
GPT-5.6 Sol	0.946	Not recorded	Not recorded	2026-07-09 · model_announcement	LLM Stats
Claude Mythos Preview	0.946	Not recorded	Not recorded	2026-04-07 · model_announcement	LLM Stats
Gemini 3.1 Pro	0.943	Not recorded	Not recorded	2026-02-19 · model_announcement	LLM Stats
Claude Opus 4.7	0.942	Not recorded	Not recorded	2026-04-16 · model_announcement	LLM Stats
Claude Opus 4.8	0.936	Not recorded	Not recorded	2026-05-28 · model_announcement	LLM Stats
GPT-5.5	0.936	Not recorded	Not recorded	2026-04-23 · model_announcement	LLM Stats
Kimi K3	0.935	Not recorded	Not recorded	2026-07-16 · model_announcement	LLM Stats
GPT-5.2 Pro	0.932	Not recorded	Not recorded	2025-12-11 · model_announcement	LLM Stats
Grok 4.5	0.93	Not recorded	Not recorded	2026-07-16 · model_announcement	LLM Stats
GPT-5.6 Terra	0.929	Not recorded	Not recorded	2026-07-09 · model_announcement	LLM Stats
GPT-5.4	0.928	Not recorded	Not recorded	2026-03-05 · model_announcement	LLM Stats
Qwen3.8 Max	0.926	Not recorded	Not recorded	2026-08-02 · model_announcement	LLM Stats
Qwen3.7 Max	0.924	Not recorded	Not recorded	2026-05-19 · model_announcement	LLM Stats
GPT-5.2	0.924	Not recorded	Not recorded	2025-12-11 · model_announcement	LLM Stats
GPT-5.6 Luna	0.923	Not recorded	Not recorded	2026-07-09 · model_announcement	LLM Stats
Gemini 3 Pro	0.919	Not recorded	Not recorded	2025-11-18 · model_announcement	LLM Stats
Claude Opus 4.6	0.913	Not recorded	Not recorded	2026-02-05 · model_announcement	LLM Stats
GLM-5.2	0.912	Not recorded	Not recorded	2026-06-16 · model_announcement	LLM Stats
Kimi K2.6	0.905	Not recorded	Not recorded	2026-04-20 · model_announcement	LLM Stats
Hy3	0.904	Not recorded	Not recorded	2026-07-06 · model_announcement	LLM Stats
Qwen3.6 Plus	0.904	Not recorded	Not recorded	2026-04-02 · model_announcement	LLM Stats
Gemini 3 Flash	0.904	Not recorded	Not recorded	2025-12-17 · model_announcement	LLM Stats
Qwen3.7-Plus	0.903	Not recorded	Not recorded	2026-05-31 · model_announcement	LLM Stats
DeepSeek-V4-Pro-Max	0.901	Not recorded	Not recorded	2026-04-23 · model_announcement	LLM Stats
Claude Sonnet 4.6	0.899	Not recorded	Not recorded	2026-02-17 · model_announcement	LLM Stats
Inkling-Small	0.895	Not recorded	Not recorded	2026-07-30 · model_announcement	LLM Stats
Muse Spark	0.895	Not recorded	Not recorded	2026-04-08 · model_announcement	LLM Stats
Qwen3.8-27B	0.892	Not recorded	Not recorded	2026-08-14 · model_announcement	LLM Stats
Solar Pro 4	0.89	Not recorded	Not recorded	2026-08-06 · model_announcement	LLM Stats
Seed 2.0 Pro	0.889	Not recorded	Not recorded	2026-02-14 · model_announcement	LLM Stats
Qwen3.5-397B-A17B	0.884	Not recorded	Not recorded	2026-02-16 · model_announcement	LLM Stats
Grok-4 Heavy	0.884	Not recorded	Not recorded	2025-07-09 · model_announcement	LLM Stats
DeepSeek-V4-Flash-Max	0.881	Not recorded	Not recorded	2026-04-23 · model_announcement	LLM Stats
GPT-5.1	0.881	Not recorded	Not recorded	2025-11-13 · model_announcement	LLM Stats
GPT-5.1 Thinking	0.881	Not recorded	Not recorded	2025-11-12 · model_announcement	LLM Stats
GPT-5.1 High	0.881	Not recorded	Not recorded	2025-11-12 · model_announcement	LLM Stats
GPT-5.1 Instant	0.881	Not recorded	Not recorded	2025-11-12 · model_announcement	LLM Stats
GPT-5 Medium	0.881	Not recorded	Not recorded	2025-08-07 · model_announcement	LLM Stats
GPT-5.4 mini	0.88	Not recorded	Not recorded	2026-03-17 · model_announcement	LLM Stats
Qwen3.6-27B	0.878	Not recorded	Not recorded	2026-04-21 · model_announcement	LLM Stats
Kimi K2.5	0.876	Not recorded	Not recorded	2026-01-27 · model_announcement	LLM Stats
Grok-4	0.875	Not recorded	Not recorded	2025-07-09 · model_announcement	LLM Stats
DeepSeek-V4-Flash-0423	0.874	Not recorded	Not recorded	2026-04-23 · model_announcement	LLM Stats
GPT-5 High	0.873	Not recorded	Not recorded	2025-08-07 · model_announcement	LLM Stats
Nemotron 3 Ultra (550B A55B)	0.87	Not recorded	Not recorded	2026-06-04 · model_announcement	LLM Stats
Claude Opus 4.5	0.87	Not recorded	Not recorded	2025-11-24 · model_announcement	LLM Stats
Gemini 3.1 Flash-Lite	0.869	Not recorded	Not recorded	2026-03-03 · model_announcement	LLM Stats
Qwen3.5-122B-A10B	0.866	Not recorded	Not recorded	2026-02-24 · model_announcement	LLM Stats
Gemini 2.5 Pro Preview 06-05	0.864	Not recorded	Not recorded	2025-06-05 · model_announcement	LLM Stats
GLM-5.1	0.862	Not recorded	Not recorded	2026-04-07 · model_announcement	LLM Stats
Qwen3.6-35B-A3B	0.86	Not recorded	Not recorded	2026-04-16 · model_announcement	LLM Stats
GLM-4.7	0.857	Not recorded	Not recorded	2025-12-22 · model_announcement	LLM Stats
Grok 4 Fast	0.857	Not recorded	Not recorded	2025-08-28 · model_announcement	LLM Stats
GPT-5	0.857	Not recorded	Not recorded	2025-08-07 · model_announcement	LLM Stats
GPT-5.5 Instant	0.856	Not recorded	Not recorded	2026-05-05 · model_announcement	LLM Stats
Qwen3.5-27B	0.855	Not recorded	Not recorded	2026-02-24 · model_announcement	LLM Stats
Seed 2.0 Lite	0.851	Not recorded	Not recorded	2026-02-14 · model_announcement	LLM Stats
ERNIE 5.0	0.85	Not recorded	Not recorded	2026-01-22 · model_announcement	LLM Stats
Claude 3.7 Sonnet	0.848	Not recorded	Not recorded	2025-02-24 · model_announcement	LLM Stats
MAI-Code-1-Flash	0.846	Not recorded	Not recorded	2026-06-02 · model_announcement	LLM Stats
Grok-3	0.846	Not recorded	Not recorded	2025-02-17 · model_announcement	LLM Stats
Kimi K2-Thinking-0905	0.845	Not recorded	Not recorded	2025-09-05 · model_announcement	LLM Stats
Gemma 4 31B	0.843	Not recorded	Not recorded	2026-04-02 · model_announcement	LLM Stats
MAI-Thinking-1	0.842	Not recorded	Not recorded	2026-06-02 · model_announcement	LLM Stats
Qwen3.5-35B-A3B	0.842	Not recorded	Not recorded	2026-02-24 · model_announcement	LLM Stats
Grok-3 Mini	0.84	Not recorded	Not recorded	2025-02-17 · model_announcement	LLM Stats
ChatGPT-4o Latest	0.84	Not recorded	Not recorded	2024-05-13 · model_announcement	LLM Stats
MiMo-V2-Flash	0.837	Not recorded	Not recorded	2025-12-16 · model_announcement	LLM Stats
Muse Glimmer-30B	0.835	Not recorded	Not recorded	2026-08-10 · model_announcement	LLM Stats
Claude Sonnet 4.5	0.834	Not recorded	Not recorded	2025-09-29 · model_announcement	LLM Stats
o3	0.833	Not recorded	Not recorded	2025-04-16 · model_announcement	LLM Stats
Gemini 2.5 Pro	0.83	Not recorded	Not recorded	2025-05-20 · model_announcement	LLM Stats
GPT-5.4 nano	0.828	Not recorded	Not recorded	2026-03-17 · model_announcement	LLM Stats
Gemini 2.5 Flash	0.828	Not recorded	Not recorded	2025-05-20 · model_announcement	LLM Stats
Nemotron 3 Super (120B A12B)	0.827	Not recorded	Not recorded	2026-03-11 · model_announcement	LLM Stats
DeepSeek-V3.2 (Thinking)	0.824	Not recorded	Not recorded	2025-12-01 · model_announcement	LLM Stats
DeepSeek-V3.2	0.824	Not recorded	Not recorded	2025-12-01 · model_announcement	LLM Stats
Gemma 4 26B-A4B	0.823	Not recorded	Not recorded	2026-04-02 · model_announcement	LLM Stats
GPT-5 mini	0.823	Not recorded	Not recorded	2025-08-07 · model_announcement	LLM Stats
Qwen3.5-9B	0.817	Not recorded	Not recorded	2026-03-02 · model_announcement	LLM Stats
LongCat-Flash-Thinking	0.815	Not recorded	Not recorded	2025-09-22 · model_announcement	LLM Stats
Nova 2 Pro	0.814	Not recorded	Not recorded	2025-12-02 · model_announcement	LLM Stats
o4-mini	0.814	Not recorded	Not recorded	2025-04-16 · model_announcement	LLM Stats
Qwen3-235B-A22B-Thinking-2507	0.811	Not recorded	Not recorded	2025-07-25 · model_announcement	LLM Stats
MiniMax M2.1	0.81	Not recorded	Not recorded	2025-12-23 · model_announcement	LLM Stats
GLM-4.6	0.81	Not recorded	Not recorded	2025-09-30 · model_announcement	LLM Stats
DeepSeek-R1-0528	0.81	Not recorded	Not recorded	2025-05-28 · model_announcement	LLM Stats
Claude Opus 4.1	0.809	Not recorded	Not recorded	2025-08-05 · model_announcement	LLM Stats
GPT OSS 120B High	0.809	Not recorded	Not recorded	2025-08-05 · model_announcement	LLM Stats
LongCat-Flash-Thinking-2601	0.805	Not recorded	Not recorded	2026-01-14 · model_announcement	LLM Stats
GPT OSS 120B	0.801	Not recorded	Not recorded	2025-08-05 · model_announcement	LLM Stats
DeepSeek-V3.2-Exp	0.799	Not recorded	Not recorded	2025-09-29 · model_announcement	LLM Stats
Nova 2 Lite	0.796	Not recorded	Not recorded	2025-12-02 · model_announcement	LLM Stats
Claude Opus 4	0.796	Not recorded	Not recorded	2025-05-22 · model_announcement	LLM Stats
GLM-4.5	0.791	Not recorded	Not recorded	2025-07-28 · model_announcement	LLM Stats
o1-pro	0.79	Not recorded	Not recorded	2024-12-17 · model_announcement	LLM Stats
Gemma 4 12B	0.788	Not recorded	Not recorded	2026-05-23 · model_announcement	LLM Stats
Sarvam-105B	0.787	Not recorded	Not recorded	2026-03-06 · model_announcement	LLM Stats
MiniMax M2	0.78	Not recorded	Not recorded	2025-10-27 · model_announcement	LLM Stats
o1	0.78	Not recorded	Not recorded	2024-12-17 · model_announcement	LLM Stats
Qwen3-235B-A22B-Instruct-2507	0.775	Not recorded	Not recorded	2025-07-22 · model_announcement	LLM Stats
Qwen3-Next-80B-A3B-Thinking	0.772	Not recorded	Not recorded	2025-09-10 · model_announcement	LLM Stats
o3-mini	0.772	Not recorded	Not recorded	2025-01-30 · model_announcement	LLM Stats
Qwen3.5-4B	0.762	Not recorded	Not recorded	2026-03-02 · model_announcement	LLM Stats
Llama 3.1 Nemotron Ultra 253B v1	0.7601	Not recorded	Not recorded	2025-04-07 · model_announcement	LLM Stats
Kimi K2 0905	0.758	Not recorded	Not recorded	2025-09-05 · model_announcement	LLM Stats
Nemotron 3.5 Lightning (30B A3B)	0.7544	Not recorded	Not recorded	2026-08-11 · model_announcement	LLM Stats
Claude Sonnet 4	0.754	Not recorded	Not recorded	2025-05-22 · model_announcement	LLM Stats
GLM-4.7-Flash	0.752	Not recorded	Not recorded	2026-01-19 · model_announcement	LLM Stats
Kimi K2-Instruct-0905	0.751	Not recorded	Not recorded	2025-09-05 · model_announcement	LLM Stats
Kimi K2 Instruct	0.751	Not recorded	Not recorded	2025-07-11 · model_announcement	LLM Stats
Nemotron 3 Nano (30B A3B)	0.75	Not recorded	Not recorded	2025-12-15 · model_announcement	LLM Stats
GLM-4.5-Air	0.75	Not recorded	Not recorded	2025-07-28 · model_announcement	LLM Stats
DeepSeek-V3.1	0.749	Not recorded	Not recorded	2025-01-10 · model_announcement	LLM Stats
Qwen3 VL 30B A3B Thinking	0.744	Not recorded	Not recorded	2025-09-22 · model_announcement	LLM Stats
GPT OSS 20B High	0.742	Not recorded	Not recorded	2025-08-05 · model_announcement	LLM Stats
Gemini 2.0 Flash Thinking	0.742	Not recorded	Not recorded	2025-01-21 · model_announcement	LLM Stats
Mercury 2	0.74	Not recorded	Not recorded	2026-02-24 · model_announcement	LLM Stats
ERNIE 4.5	0.74	Not recorded	Not recorded	2025-06-25 · model_announcement	LLM Stats
DeepSeek R1 Zero	0.733	Not recorded	Not recorded	2025-01-20 · model_announcement	LLM Stats
o1-preview	0.733	Not recorded	Not recorded	2024-09-12 · model_announcement	LLM Stats
LongCat-Flash-Chat	0.7323	Not recorded	Not recorded	2025-08-29 · model_announcement	LLM Stats
DiffusionGemma 26B-A4B	0.732	Not recorded	Not recorded	2026-06-10 · model_announcement	LLM Stats
Qwen3 VL 32B Thinking	0.731	Not recorded	Not recorded	2025-09-22 · model_announcement	LLM Stats
Claude Haiku 4.5	0.73	Not recorded	Not recorded	2025-10-15 · model_announcement	LLM Stats
Qwen3-Next-80B-A3B-Instruct	0.729	Not recorded	Not recorded	2025-09-10 · model_announcement	LLM Stats
GPT OSS 20B	0.715	Not recorded	Not recorded	2025-08-05 · model_announcement	LLM Stats
Mistral Small 4	0.712	Not recorded	Not recorded	2026-03-16 · model_announcement	LLM Stats
Ministral 3 (14B Reasoning 2512)	0.712	Not recorded	Not recorded	2025-12-04 · model_announcement	LLM Stats
GPT-5 nano	0.712	Not recorded	Not recorded	2025-08-07 · model_announcement	LLM Stats
Magistral Medium	0.708	Not recorded	Not recorded	2025-06-10 · model_announcement	LLM Stats
Qwen3 VL 30B A3B Instruct	0.704	Not recorded	Not recorded	2025-09-22 · model_announcement	LLM Stats
GPT-4o	0.701	Not recorded	Not recorded	2024-08-06 · model_announcement	LLM Stats
MiniMax M1 80K	0.7	Not recorded	Not recorded	2025-06-16 · model_announcement	LLM Stats
Qwen3 VL 8B Thinking	0.699	Not recorded	Not recorded	2025-09-22 · model_announcement	LLM Stats
Llama 4 Maverick	0.698	Not recorded	Not recorded	2025-04-05 · model_announcement	LLM Stats
GPT-4.5	0.695	Not recorded	Not recorded	2025-02-27 · model_announcement	LLM Stats
MiniMax M1 40K	0.692	Not recorded	Not recorded	2025-06-16 · model_announcement	LLM Stats
Qwen3 VL 32B Instruct	0.689	Not recorded	Not recorded	2025-09-22 · model_announcement	LLM Stats
Phi 4 Reasoning Plus	0.689	Not recorded	Not recorded	2025-04-30 · model_announcement	LLM Stats
DeepSeek-V3 0324	0.684	Not recorded	Not recorded	2025-03-25 · model_announcement	LLM Stats
Magistral Small 2506	0.6818	Not recorded	Not recorded	2025-06-10 · model_announcement	LLM Stats
Claude 3.5 Sonnet	0.672	Not recorded	Not recorded	2024-10-22 · model_announcement	LLM Stats
Ministral 3 (8B Reasoning 2512)	0.668	Not recorded	Not recorded	2025-12-04 · model_announcement	LLM Stats
LongCat-Flash-Lite	0.6678	Not recorded	Not recorded	2026-02-05 · model_announcement	LLM Stats
MiMo-V2.5-Pro	0.667	Not recorded	Not recorded	2026-04-27 · model_announcement	LLM Stats
Llama-3.3 Nemotron Super 49B v1	0.6667	Not recorded	Not recorded	2025-03-18 · model_announcement	LLM Stats
Sarvam-30B	0.665	Not recorded	Not recorded	2026-03-06 · model_announcement	LLM Stats
GPT-4.1	0.663	Not recorded	Not recorded	2025-04-14 · model_announcement	LLM Stats
Hermes 3 70B	0.661	Not recorded	Not recorded	2024-08-15 · model_announcement	LLM Stats
Phi 4 Reasoning	0.658	Not recorded	Not recorded	2025-04-30 · model_announcement	LLM Stats
Qwen3 30B A3B	0.658	Not recorded	Not recorded	2025-04-29 · model_announcement	LLM Stats
QwQ-32B	0.652	Not recorded	Not recorded	2025-03-05 · model_announcement	LLM Stats
DeepSeek R1 Distill Llama 70B	0.652	Not recorded	Not recorded	2025-01-20 · model_announcement	LLM Stats
QwQ-32B-Preview	0.652	Not recorded	Not recorded	2024-11-28 · model_announcement	LLM Stats
GPT-4.1 mini	0.65	Not recorded	Not recorded	2025-04-14 · model_announcement	LLM Stats
Gemini 2.5 Flash-Lite	0.646	Not recorded	Not recorded	2025-06-17 · model_announcement	LLM Stats
Qwen3 VL 4B Thinking	0.641	Not recorded	Not recorded	2025-09-22 · model_announcement	LLM Stats
Nemotron Nano 9B v2	0.64	Not recorded	Not recorded	2025-08-18 · model_announcement	LLM Stats
DeepSeek R1 Distill Qwen 32B	0.621	Not recorded	Not recorded	2025-01-20 · model_announcement	LLM Stats
Gemini 2.0 Flash	0.621	Not recorded	Not recorded	2024-12-01 · model_announcement	LLM Stats
Qwen3 Max	0.62	Not recorded	Not recorded	2025-12-15 · model_announcement	LLM Stats
o1-mini	0.6	Not recorded	Not recorded	2024-09-12 · model_announcement	LLM Stats
Claude 3.5 Sonnet	0.594	Not recorded	Not recorded	2024-06-21 · model_announcement	LLM Stats
DeepSeek R1 Distill Qwen 14B	0.591	Not recorded	Not recorded	2025-01-20 · model_announcement	LLM Stats
DeepSeek-V3	0.591	Not recorded	Not recorded	2024-12-25 · model_announcement	LLM Stats
Gemini 1.5 Pro	0.591	Not recorded	Not recorded	2024-05-01 · model_announcement	LLM Stats
Gemma 4 E4B	0.586	Not recorded	Not recorded	2026-04-02 · model_announcement	LLM Stats
Llama 4 Scout	0.572	Not recorded	Not recorded	2025-04-05 · model_announcement	LLM Stats
Phi 4	0.561	Not recorded	Not recorded	2024-12-12 · model_announcement	LLM Stats
Grok-2	0.56	Not recorded	Not recorded	2024-08-13 · model_announcement	LLM Stats
Llama 3.1 Nemotron Nano 8B V1	0.541	Not recorded	Not recorded	2025-03-18 · model_announcement	LLM Stats
GPT-4o	0.536	Not recorded	Not recorded	2024-05-13 · model_announcement	LLM Stats
Min istral 3 (3B Reasoning 2512)	0.534	Not recorded	Not recorded	2025-12-04 · model_announcement	LLM Stats
Phi 4 Mini Reasoning	0.52	Not recorded	Not recorded	2025-04-30 · model_announcement	LLM Stats
Qwen3.5-2B	0.516	Not recorded	Not recorded	2026-03-02 · model_announcement	LLM Stats
Gemini 2.0 Flash-Lite	0.515	Not recorded	Not recorded	2025-02-05 · model_announcement	LLM Stats
Grok-2 mini	0.51	Not recorded	Not recorded	2024-08-13 · model_announcement	LLM Stats
Gemini 1.5 Flash	0.51	Not recorded	Not recorded	2024-05-01 · model_announcement	LLM Stats
Llama 3.1 405B Instruct	0.507	Not recorded	Not recorded	2024-07-23 · model_announcement	LLM Stats
Llama 3.3 70B Instruct	0.505	Not recorded	Not recorded	2024-12-06 · model_announcement	LLM Stats
Claude 3 Opus	0.504	Not recorded	Not recorded	2024-02-29 · model_announcement	LLM Stats
GPT-4.1 nano	0.503	Not recorded	Not recorded	2025-04-14 · model_announcement	LLM Stats
Qwen2.5 32B Instruct	0.495	Not recorded	Not recorded	2024-09-19 · model_announcement	LLM Stats
DeepSeek R1 Distill Qwen 7B	0.491	Not recorded	Not recorded	2025-01-20 · model_announcement	LLM Stats
DeepSeek R1 Distill Llama 8B	0.49	Not recorded	Not recorded	2025-01-20 · model_announcement	LLM Stats
Qwen2.5 72B Instruct	0.49	Not recorded	Not recorded	2024-09-19 · model_announcement	LLM Stats
Kimi K2 Base	0.481	Not recorded	Not recorded	2025-07-11 · model_announcement	LLM Stats
GPT-4 Turbo	0.48	Not recorded	Not recorded	2024-04-09 · model_announcement	LLM Stats
Qwen3 235B A22B	0.4747	Not recorded	Not recorded	2025-04-29 · model_announcement	LLM Stats
Nova Pro	0.469	Not recorded	Not recorded	2024-11-20 · model_announcement	LLM Stats
Llama 3.2 90B Instruct	0.467	Not recorded	Not recorded	2024-09-25 · model_announcement	LLM Stats
Mistral Small 3.2 24B Instruct	0.4613	Not recorded	Not recorded	2025-06-20 · model_announcement	LLM Stats
Qwen2.5 VL 32B Instruct	0.46	Not recorded	Not recorded	2025-02-28 · model_announcement	LLM Stats
Mistral Small 3.1 24B Instruct	0.4596	Not recorded	Not recorded	2025-03-17 · model_announcement	LLM Stats
Qwen2.5 14B Instruct	0.455	Not recorded	Not recorded	2024-09-19 · model_announcement	LLM Stats
Mistral Small 3 24B Instruct	0.453	Not recorded	Not recorded	2025-01-30 · model_announcement	LLM Stats
Mistral Large 3 (675B Instruct 2512 NVFP4)	0.439	Not recorded	Not recorded	2025-12-04 · model_announcement	LLM Stats
Mistral Large 3 (675B Instruct 2512)	0.439	Not recorded	Not recorded	2025-12-04 · model_announcement	LLM Stats
Mistral Large 3 (675B Instruct 2512 Eagle)	0.439	Not recorded	Not recorded	2025-12-04 · model_announcement	LLM Stats
Mistral Large 3 (675B Base)	0.439	Not recorded	Not recorded	2025-12-04 · model_announcement	LLM Stats
Gemma 4 E2B	0.434	Not recorded	Not recorded	2026-04-02 · model_announcement	LLM Stats
Gemma 3 27B	0.424	Not recorded	Not recorded	2025-03-12 · model_announcement	LLM Stats
Qwen2 72B Instruct	0.424	Not recorded	Not recorded	2024-07-23 · model_announcement	LLM Stats
Nova Lite	0.42	Not recorded	Not recorded	2024-11-20 · model_announcement	LLM Stats
Llama 3.1 70B Instruct	0.417	Not recorded	Not recorded	2024-07-23 · model_announcement	LLM Stats
Claude 3.5 Haiku	0.416	Not recorded	Not recorded	2024-10-22 · model_announcement	LLM Stats
Gemma 3 12B	0.409	Not recorded	Not recorded	2025-03-12 · model_announcement	LLM Stats
Gemini Diffusion	0.404	Not recorded	Not recorded	2025-05-20 · model_announcement	LLM Stats
Claude 3 Sonnet	0.404	Not recorded	Not recorded	2024-02-29 · model_announcement	LLM Stats
GPT-4o mini	0.402	Not recorded	Not recorded	2024-07-18 · model_announcement	LLM Stats
Nova Micro	0.4	Not recorded	Not recorded	2024-11-20 · model_announcement	LLM Stats
Gemini 1.5 Flash 8B	0.384	Not recorded	Not recorded	2024-03-15 · model_announcement	LLM Stats
Mistral Small 3.1 24B Base	0.375	Not recorded	Not recorded	2025-03-17 · model_announcement	LLM Stats
Jamba 1.5 Large	0.369	Not recorded	Not recorded	2024-08-22 · model_announcement	LLM Stats
Phi-3.5-MoE-instruct	0.368	Not recorded	Not recorded	2024-08-23 · model_announcement	LLM Stats
Qwen2.5 7B Instruct	0.364	Not recorded	Not recorded	2024-09-19 · model_announcement	LLM Stats
Grok-1.5	0.359	Not recorded	Not recorded	2024-03-28 · model_announcement	LLM Stats
GPT-4	0.357	Not recorded	Not recorded	2023-06-13 · model_announcement	LLM Stats
Mistral Small 3 24B Base	0.3437	Not recorded	Not recorded	2025-01-30 · model_announcement	LLM Stats
DeepSeek R1 Distill Qwen 1.5B	0.338	Not recorded	Not recorded	2025-01-20 · model_announcement	LLM Stats
Claude 3 Haiku	0.333	Not recorded	Not recorded	2024-03-13 · model_announcement	LLM Stats
Llama 3.2 11B Instruct	0.328	Not recorded	Not recorded	2024-09-25 · model_announcement	LLM Stats
Llama 3.2 3B Instruct	0.328	Not recorded	Not recorded	2024-09-25 · model_announcement	LLM Stats
Jamba 1.5 Mini	0.323	Not recorded	Not recorded	2024-08-22 · model_announcement	LLM Stats
Qwen2.5-Omni-7B	0.308	Not recorded	Not recorded	2025-03-27 · model_announcement	LLM Stats
Gemma 3 4B	0.308	Not recorded	Not recorded	2025-03-12 · model_announcement	LLM Stats
GPT-3.5 Turbo	0.308	Not recorded	Not recorded	2023-03-21 · model_announcement	LLM Stats
Phi-3.5-mini-instruct	0.304	Not recorded	Not recorded	2024-08-23 · model_announcement	LLM Stats
Llama 3.1 8B Instruct	0.304	Not recorded	Not recorded	2024-07-23 · model_announcement	LLM Stats
Gemini 1.0 Pro	0.279	Not recorded	Not recorded	2024-02-15 · model_announcement	LLM Stats
Qwen2 7B Instruct	0.253	Not recorded	Not recorded	2024-07-23 · model_announcement	LLM Stats
Phi 4 Mini	0.252	Not recorded	Not recorded	2025-02-01 · model_announcement	LLM Stats
Gemma 3n E2B Instructed	0.248	Not recorded	Not recorded	2025-06-26 · model_announcement	LLM Stats
Gemma 3n E2B Instructed LiteRT (Preview)	0.248	Not recorded	Not recorded	2025-05-20 · model_announcement	LLM Stats
Gemma 3n E4B Instructed	0.237	Not recorded	Not recorded	2025-06-26 · model_announcement	LLM Stats
Gemma 3n E4B Instructed LiteRT Preview	0.237	Not recorded	Not recorded	2025-05-20 · model_announcement	LLM Stats
Gemma 3 1B	0.192	Not recorded	Not recorded	2025-03-12 · model_announcement	LLM Stats
Qwen3.5-0.8B	0.119	Not recorded	Not recorded	2026-03-02 · model_announcement	LLM Stats

Related records in other sources

GPQA

Specialist science questions

Main, Extended and Diamond

Accuracy needs an evaluation setup

What Google-proof means here

Small datasets and exposure

Evidence available for this benchmark

Start with these resources

Recorded attention

Evaluation context and evidence

Reported results 239

Explore related benchmarks

Put this benchmark in context

Six Navers Lab / Einsia benchmarks and the work they test

AI benchmarks and leaderboards are different things

How to compare AI benchmark scores without mixing protocols

Four new agent benchmarks: adaptation, safety and video

Continue exploring

Related records in other sources