LLM Stats

HumanEval

HumanEval asks a model to implement a small Python function from a specification, then checks its behavior by running tests. Read pass@1 and pass@k separately.

Catalog source

AI Benchmark Radar / Equality Machine · Reviewed 2026-10-25

Original 164-task Python HumanEval with its original tests. HumanEval+, multilingual variants and infilling variants remain separate evaluations.

Function completion from a specification

A prompt supplies a function signature and docstring. The original set contains 164 hand-written programming problems with tests, spanning language understanding, algorithms and elementary mathematics. The model supplies code that satisfies the requested behavior.

Evaluating Large Language Models Trained on Code · HumanEval and pass@k

Execution determines correctness

The harness executes each generated completion against the task's tests and records pass or failure. Matching reference wording or source code is unnecessary. Runtime errors and timeouts are failures under the harness configuration.

OpenAI HumanEval · Official evaluation harness · HumanEval · Functional evaluation and pass@k implementation

What pass@k measures

Pass@k estimates whether at least one of k generated candidates passes. The official estimator uses the number of samples and successful samples per problem. A high pass@100 does not mean a user receives correct code on the first attempt.

HumanEval · Functional evaluation and pass@k implementation

Match the sampling and test version

Compare the same k, sampling temperature, completion handling and test suite. The paper shows that preferred temperatures differ with k. Results from extra tests or other languages should retain their own variant labels.

Evaluating Large Language Models Trained on Code · HumanEval and pass@k · OpenAI HumanEval · Official evaluation harness

A compact coding signal

Passing a finite test suite is evidence about those cases, not a proof of all behavior. These short functions do not cover repository maintenance or long projects. The official repository now marks the original harness as unmaintained historical material.

OpenAI HumanEval · Official evaluation harness · Evaluating Large Language Models Trained on Code · HumanEval and pass@k

Data source and credit: LLM Stats

Evidence available for this benchmark

This source record contains 66 numeric observations. The source distinguishes 66 model identities or configurations. A model may appear more than once; check the version, tools and run budget before comparing. Read results and row-level evidence →

Metric direction: higher is better

Start with these resources

Paper · task design and experiments · arxiv.org
Code · implementation and instructions · github.com
Catalog source · api.zeroeval.com

Cited documents and dates (1)

HumanEval

Catalog collected on 2026-08-17; resources and fields reflect that saved source.

Recorded attention

Observed on: 2026-10-12

GitHub stars: 3,402
Hugging Face paper upvotes: 12

openai/human-eval · View source snapshot

Dated attention observations, not live counts or benchmark scores.

Evaluation context and evidence

Source: LLM Stats
Publisher: Not recorded
Benchmark release: 2021-07-08
Collected: 2026-08-17
Modality: text

These are source-reported observations, not independent reruns by this site. Check the task version, evaluation protocol, unit and reporting date before comparing models. Results from different setups are not a universal ranking.

Reported results 66

Shown in source observation order, not ranked. Dates retain the precision provided by the source.

Model	Value	Unit	Protocol	Reported date and basis	Source
MiniCPM-SALA	0.9512	Not recorded	Not recorded	2026-02-11 · model_announcement	LLM Stats
Kimi K2 0905	0.945	Not recorded	Not recorded	2025-09-05 · model_announcement	LLM Stats
Claude 3.5 Sonnet	0.937	Not recorded	Not recorded	2024-10-22 · model_announcement	LLM Stats
GPT-5	0.934	Not recorded	Not recorded	2025-08-07 · model_announcement	LLM Stats
Kimi K2 Instruct	0.933	Not recorded	Not recorded	2025-07-11 · model_announcement	LLM Stats
Qwen2.5-Coder 32B Instruct	0.927	Not recorded	Not recorded	2024-09-19 · model_announcement	LLM Stats
o1-mini	0.924	Not recorded	Not recorded	2024-09-12 · model_announcement	LLM Stats
Sarvam-30B	0.921	Not recorded	Not recorded	2026-03-06 · model_announcement	LLM Stats
Mistral Large 2	0.92	Not recorded	Not recorded	2024-07-24 · model_announcement	LLM Stats
Claude 3.5 Sonnet	0.92	Not recorded	Not recorded	2024-06-21 · model_announcement	LLM Stats
Qwen2.5 VL 32B Instruct	0.915	Not recorded	Not recorded	2025-02-28 · model_announcement	LLM Stats
GPT-4o	0.902	Not recorded	Not recorded	2024-05-13 · model_announcement	LLM Stats
Granite 3.3 8B Instruct	0.8973	Not recorded	Not recorded	2025-04-16 · model_announcement	LLM Stats
Granite 3.3 8B Base	0.8973	Not recorded	Not recorded	2025-04-16 · model_announcement	LLM Stats
Gemini Diffusion	0.896	Not recorded	Not recorded	2025-05-20 · model_announcement	LLM Stats
Nova Pro	0.89	Not recorded	Not recorded	2024-11-20 · model_announcement	LLM Stats
Llama 3.1 405B Instruct	0.89	Not recorded	Not recorded	2024-07-23 · model_announcement	LLM Stats
DeepSeek-V2.5	0.89	Not recorded	Not recorded	2024-05-08 · model_announcement	LLM Stats
LongCat-Flash-Chat	0.8841	Not recorded	Not recorded	2025-08-29 · model_announcement	LLM Stats
Mistral Small 3.1 24B Instruct	0.8841	Not recorded	Not recorded	2025-03-17 · model_announcement	LLM Stats
Llama 3.3 70B Instruct	0.884	Not recorded	Not recorded	2024-12-06 · model_announcement	LLM Stats
Qwen2.5-Coder 7B Instruct	0.884	Not recorded	Not recorded	2024-09-19 · model_announcement	LLM Stats
Qwen2.5 32B Instruct	0.884	Not recorded	Not recorded	2024-09-19 · model_announcement	LLM Stats
Grok-2	0.884	Not recorded	Not recorded	2024-08-13 · model_announcement	LLM Stats
o1	0.881	Not recorded	Not recorded	2024-12-17 · model_announcement	LLM Stats
Claude 3.5 Haiku	0.881	Not recorded	Not recorded	2024-10-22 · model_announcement	LLM Stats
GPT-4.5	0.88	Not recorded	Not recorded	2025-02-27 · model_announcement	LLM Stats
Gemma 3 27B	0.878	Not recorded	Not recorded	2025-03-12 · model_announcement	LLM Stats
GPT-4o mini	0.872	Not recorded	Not recorded	2024-07-18 · model_announcement	LLM Stats
GPT-4 Turbo	0.871	Not recorded	Not recorded	2024-04-09 · model_announcement	LLM Stats
Qwen2.5 72B Instruct	0.866	Not recorded	Not recorded	2024-09-19 · model_announcement	LLM Stats
Qwen2 72B Instruct	0.86	Not recorded	Not recorded	2024-07-23 · model_announcement	LLM Stats
Grok-2 mini	0.857	Not recorded	Not recorded	2024-08-13 · model_announcement	LLM Stats
Gemma 3 12B	0.854	Not recorded	Not recorded	2025-03-12 · model_announcement	LLM Stats
Nova Lite	0.854	Not recorded	Not recorded	2024-11-20 · model_announcement	LLM Stats
Claude 3 Opus	0.849	Not recorded	Not recorded	2024-02-29 · model_announcement	LLM Stats
Mistral Small 3 24B Instruct	0.848	Not recorded	Not recorded	2025-01-30 · model_announcement	LLM Stats
Qwen2.5 7B Instruct	0.848	Not recorded	Not recorded	2024-09-19 · model_announcement	LLM Stats
Gemini 1.5 Pro	0.841	Not recorded	Not recorded	2024-05-01 · model_announcement	LLM Stats
Qwen2.5 14B Instruct	0.835	Not recorded	Not recorded	2024-09-19 · model_announcement	LLM Stats
Phi 4	0.826	Not recorded	Not recorded	2024-12-12 · model_announcement	LLM Stats
IBM Granite 4.0 Tiny Preview	0.824	Not recorded	Not recorded	2025-05-02 · model_announcement	LLM Stats
Nova Micro	0.811	Not recorded	Not recorded	2024-11-20 · model_announcement	LLM Stats
Codestral-22B	0.811	Not recorded	Not recorded	2024-05-29 · model_announcement	LLM Stats
Llama 3.1 70B Instruct	0.805	Not recorded	Not recorded	2024-07-23 · model_announcement	LLM Stats
Qwen2 7B Instruct	0.799	Not recorded	Not recorded	2024-07-23 · model_announcement	LLM Stats
Qwen2.5-Omni-7B	0.787	Not recorded	Not recorded	2025-03-27 · model_announcement	LLM Stats
Claude 3 Haiku	0.759	Not recorded	Not recorded	2024-03-13 · model_announcement	LLM Stats
Gemma 3n E4B Instructed	0.75	Not recorded	Not recorded	2025-06-26 · model_announcement	LLM Stats
Gemma 3n E4B Instructed LiteRT Preview	0.75	Not recorded	Not recorded	2025-05-20 · model_announcement	LLM Stats
Gemini 1.5 Flash	0.743	Not recorded	Not recorded	2024-05-01 · model_announcement	LLM Stats
Grok-1.5	0.741	Not recorded	Not recorded	2024-03-28 · model_announcement	LLM Stats
Claude 3 Sonnet	0.73	Not recorded	Not recorded	2024-02-29 · model_announcement	LLM Stats
Llama 3.1 8B Instruct	0.726	Not recorded	Not recorded	2024-07-23 · model_announcement	LLM Stats
Pixtral-12B	0.72	Not recorded	Not recorded	2024-09-17 · model_announcement	LLM Stats
Gemma 3 4B	0.713	Not recorded	Not recorded	2025-03-12 · model_announcement	LLM Stats
Phi-3.5-MoE-instruct	0.707	Not recorded	Not recorded	2024-08-23 · model_announcement	LLM Stats
GPT-3.5 Turbo	0.68	Not recorded	Not recorded	2023-03-21 · model_announcement	LLM Stats
GPT-4	0.67	Not recorded	Not recorded	2023-06-13 · model_announcement	LLM Stats
Gemma 3n E2B Instructed	0.665	Not recorded	Not recorded	2025-06-26 · model_announcement	LLM Stats
Gemma 3n E2B Instructed LiteRT (Preview)	0.665	Not recorded	Not recorded	2025-05-20 · model_announcement	LLM Stats
Phi-3.5-mini-instruct	0.628	Not recorded	Not recorded	2024-08-23 · model_announcement	LLM Stats
Gemma 2 27B	0.518	Not recorded	Not recorded	2024-06-27 · model_announcement	LLM Stats
Gemma 3 1B	0.415	Not recorded	Not recorded	2025-03-12 · model_announcement	LLM Stats
Gemma 2 9B	0.402	Not recorded	Not recorded	2024-06-27 · model_announcement	LLM Stats
Ministral 8B Instruct	0.348	Not recorded	Not recorded	2024-10-16 · model_announcement	LLM Stats

Related records in other sources

HumanEval

Function completion from a specification

Execution determines correctness

What pass@k measures

Match the sampling and test version

A compact coding signal

Evidence available for this benchmark

Start with these resources

Recorded attention

Evaluation context and evidence

Reported results 66

Explore related benchmarks

Put this benchmark in context

Six Navers Lab / Einsia benchmarks and the work they test

AI benchmarks and leaderboards are different things

How to compare AI benchmark scores without mixing protocols

Four new agent benchmarks: adaptation, safety and video

Continue exploring

Related records in other sources