AI EVALUATION, IN CONTEXT.BY EQUALITY MACHINE
Benchmark Radar.Benchmarks. Model results. Original sources.
/

Model reports

GPQA Diamond

GPQA tests difficult specialist science questions with multiple-choice answers. Diamond is a selected subset, so identifying the exact split is more useful than comparing the benchmark name alone.

Catalog source

AI Benchmark Radar / Equality Machine · Reviewed 2026-10-25

The original GPQA family, including the main, Extended and Diamond sets. A shared paper does not establish that two reported scores use the same split.

Specialist science questions

Experts wrote and validated questions in biology, chemistry and physics. Each question has a correct option and distractors. The task tests choosing the answer to a specialized problem, rather than performing a laboratory experiment or producing a research contribution.

GPQA paper · Dataset splits and limitations · GPQA · Author dataset card and exposure notice

Main, Extended and Diamond

The paper distinguishes 546 Extended questions, 448 main questions and 198 Diamond questions. Diamond applies stricter expert-agreement and non-expert-difficulty criteria. A label such as full GPQA is insufficient: check the dataset configuration or reported question count.

GPQA paper · Dataset splits and limitations

Accuracy needs an evaluation setup

Accuracy measures the fraction of selected answers that are correct. The released baselines include different prompting and open-book configurations. Keep answer extraction, examples, search access and repeat policy visible when comparing scores from separate reports.

GPQA · Official baselines and prompts

What Google-proof means here

The name reflects difficulty for skilled non-experts even when they could search the web during construction. It is not a guarantee against training contamination, nor evidence that retrieval never helps a later system.

GPQA · Author dataset card and exposure notice

Small datasets and exposure

The authors flag limited statistical power for small accuracy differences. The dataset page also asks users not to republish questions to reduce training leakage. Compare uncertainty and failure cases, and avoid interpreting a small lead as general scientific competence.

GPQA paper · Dataset splits and limitations · GPQA · Author dataset card and exposure notice

Evidence available for this benchmark

This source record contains 33 numeric observations. The source distinguishes 31 model identities or configurations. A model may appear more than once; check the version, tools and run budget before comparing. Read results and row-level evidence →

Reported metric
accuracy
Recorded unit
percent
Metric direction
higher is better

Start with these resources

Cited documents and dates (30)

Recorded attention

Observed on: 2026-10-10

GitHub stars
540
Hugging Face paper upvotes
38

View source snapshot

Dated attention observations, not live counts or benchmark scores.

Evaluation context and evidence

Source
Model reports
Publisher
Not recorded
Benchmark release
2023-11-20
Collected
Not recorded
Modality
Not recorded

These are source-reported observations, not independent reruns by this site. Check the task version, evaluation protocol, unit and reporting date before comparing models. Results from different setups are not a universal ranking.

Reported results 33

Shown in source observation order, not ranked. Dates retain the precision provided by the source.

ModelValueUnitProtocolReported date and basisSource
Claude Opus 479.6percentextended thinking up to 64K tokens, no parallel test-time compute2025-05-22 · document_publicationClaude Opus 4 and Sonnet 4
Claude Sonnet 475.4percentextended thinking up to 64K tokens, no parallel test-time compute2025-05-22 · document_publicationClaude Opus 4 and Sonnet 4
Seed2.0 Pro88.9percentas printed in Table 3 (Large Models)2026-02-14 · document_publicationSeed2.0 (Pro / Lite / Mini)
Seed2.0 Lite66.1percentas printed in Table 4 (Efficient Models)2026-02-14 · document_publicationSeed2.0 (Pro / Lite / Mini)
Seed2.0 Mini58.1percentas printed in Table 4 (Efficient Models)2026-02-14 · document_publicationSeed2.0 (Pro / Lite / Mini)
DeepSeek-R171.5percentPass@1, averaged over k=64 samples, temperature 0.6, top-p 0.95, 32K output limit2025-01-22 · document_publicationDeepSeek-R1
DeepSeek-V359.1percentPass@1, 8K output limit2024-12-27 · document_publicationDeepSeek-V3
DeepSeek-V4.1-Flash90.9percentthink max, Pass@12026-09-10 · document_publicationDeepSeek-V4.1-Flash
DeepSeek-V4-Pro72.9percentnon-think, pass@12026-04-22 · document_publicationDeepSeek-V4-Pro
DeepSeek-V4-Pro89.1percentthink high, pass@12026-04-22 · document_publicationDeepSeek-V4-Pro
DeepSeek-V4-Pro90.1percentthink max, pass@12026-04-22 · document_publicationDeepSeek-V4-Pro
DeepSeek-V4-Flash88.1percentthink max, pass@12026-06-25 · document_publicationDeepSeek-V4
Claude 4 Opus79.6percentsingle attempt, pass@12025-06-17 · document_publicationGemini 2.5 Pro
Claude 4 Sonnet75.4percentsingle attempt, pass@12025-06-17 · document_publicationGemini 2.5 Pro
Gemini 2.5 Pro86.4percentsingle attempt, pass@12025-06-17 · document_publicationGemini 2.5 Pro
Grok 3 Beta Extended Thinking80.2percentsingle attempt, pass@12025-06-17 · document_publicationGemini 2.5 Pro
o3 high83.3percentsingle attempt, pass@12025-06-17 · document_publicationGemini 2.5 Pro
Gemini 3.1 Pro94.3percentThinking (High), No tools2026-02-19 · document_publicationGemini 3.1 Pro
Gemma 4 (31B)84.3percentas reported2026-03-11 · document_publicationGemma 4 (31B)
Bespoke-Stratos-7B37.8percentPhi-4-mini-reasoning model card comparison table; Appendix A discusses AIME 2024, maximum sequence length 32768 and common temperature without specifying its value; matching per-row sampling and prompts not established; row-specific unknown setup for Bespoke-Stratos-7B; starred source row with unexplained asterisk2025-04-30 · document_publicationPhi-4-mini-reasoning (3.8B)
DeepSeek-R1-Distill-Llama-8B47.3percentPhi-4-mini-reasoning model card comparison table; Appendix A discusses AIME 2024, maximum sequence length 32768 and common temperature without specifying its value; matching per-row sampling and prompts not established; row-specific unknown setup for DeepSeek-R1-Distill-Llama-8B; unstarred source row2025-04-30 · document_publicationPhi-4-mini-reasoning (3.8B)
DeepSeek-R1-Distill-Qwen-7B49.5percentPhi-4-mini-reasoning model card comparison table; Appendix A discusses AIME 2024, maximum sequence length 32768 and common temperature without specifying its value; matching per-row sampling and prompts not established; row-specific unknown setup for DeepSeek-R1-Distill-Qwen-7B; unstarred source row2025-04-30 · document_publicationPhi-4-mini-reasoning (3.8B)
Llama-3.2-3B-Instruct25.3percentPhi-4-mini-reasoning model card comparison table; Appendix A discusses AIME 2024, maximum sequence length 32768 and common temperature without specifying its value; matching per-row sampling and prompts not established; row-specific unknown setup for Llama-3.2-3B-Instruct; unstarred source row2025-04-30 · document_publicationPhi-4-mini-reasoning (3.8B)
OpenThinker-7B42.4percentPhi-4-mini-reasoning model card comparison table; Appendix A discusses AIME 2024, maximum sequence length 32768 and common temperature without specifying its value; matching per-row sampling and prompts not established; row-specific unknown setup for OpenThinker-7B; starred source row with unexplained asterisk2025-04-30 · document_publicationPhi-4-mini-reasoning (3.8B)
Phi-4-Mini (base model, 3.8B)36.9percentPhi-4-mini-reasoning model card comparison table; Appendix A discusses AIME 2024, maximum sequence length 32768 and common temperature without specifying its value; matching per-row sampling and prompts not established; row-specific unknown setup for Phi-4-Mini (base model, 3.8B); unstarred source row2025-04-30 · document_publicationPhi-4-mini-reasoning (3.8B)
Phi-4-mini-reasoning (3.8B)52.0percentPhi-4-mini-reasoning model card comparison table; Appendix A discusses AIME 2024, maximum sequence length 32768 and common temperature without specifying its value; matching per-row sampling and prompts not established; row-specific unknown setup for Phi-4-mini-reasoning (3.8B); unstarred source row2025-04-30 · document_publicationPhi-4-mini-reasoning (3.8B)
o1-mini60.0percentPhi-4-mini-reasoning model card comparison table; Appendix A discusses AIME 2024, maximum sequence length 32768 and common temperature without specifying its value; matching per-row sampling and prompts not established; row-specific unknown setup for o1-mini; starred source row with unexplained asterisk2025-04-30 · document_publicationPhi-4-mini-reasoning (3.8B)
Kimi K393.5percentreasoning=max, top-p 0.952026-06-13 · document_publicationKimi K3
Qwen3-235B-A22B (Thinking)71.1percentaveraged over 10 samples2025-05-14 · document_publicationQwen3
Hy4 preview92.3percentas printed in Table 3 (Large Models)2026-08-28 · document_publicationHy4 preview
GLM-5.186.2percentas reported2026-04-03 · document_publicationGLM-5.1
GLM-5.291.2percentreasoning-tasks settings, temp 1.0, top_p 0.95, max gen 163840 tokens2026-06-16 · document_publicationGLM-5.2
GLM-586.0percentreasoning-tasks settings, temp 1.0, top_p 0.952026-02-11 · document_publicationGLM-5

Explore related benchmarks

Suggestions based on shared evidence or source categories, not verified equivalents.

GPQA (full)Shared paper, code or dataset →MedChemBench (Internal)Shared source categories →FrontierCSShared source categories →CritPtShared source categories →Gsci-FactShared source categories →GPQA DiamondShared source categories →CMT-BenchmarkShared source categories →SciCodeShared source categories →

Put this benchmark in context

Science benchmarks

Continue exploring

Search the catalog using this record’s source labels. Search matches are not verified equivalents.

science →Model reports · all records →