LLM Stats

IFEval

IFEval checks concrete requirements such as output format, length or required words. It is useful for separating measurable instruction compliance from whether an answer is informative or correct.

Catalog source

AI Benchmark Radar / Equality Machine · Reviewed 2026-10-25

Original Instruction-Following Evaluation with verifiable constraints and its four strict/loose, prompt/instruction metrics; not a general answer-quality judge.

Constraints that a program can check

The paper defines 25 instruction types and around 500 prompts, each containing one or more requirements. A response may need to meet several constraints simultaneously, such as a word-count condition and a specified ending.

Instruction-Following Evaluation for Large Language Models · Metrics and limitations

Checkers replace a subjective judge

The official evaluator retrieves each instruction's checking function and applies it to the response. It records both individual pass/fail decisions and whether all instructions pass. Empty responses fail the checks rather than receiving assumed compliance.

Google Research · IFEval strict and loose checking implementation

Prompt-level versus instruction-level

Prompt-level accuracy requires every constraint in a prompt to pass. Instruction-level accuracy counts constraints individually. A model can therefore improve individual compliance while still failing many complete prompts; report which denominator the number uses.

Instruction-Following Evaluation for Large Language Models · Metrics and limitations · Google Research · IFEval strict and loose checking implementation

Strict and loose are different checks

Strict checking uses the original response. Loose checking also tries prescribed transformations, such as removing formatting or introductory lines. This can recover false negatives, but may create false positives; it complements rather than replaces the strict result.

Instruction-Following Evaluation for Large Language Models · Metrics and limitations · Google Research · IFEval strict and loose checking implementation

Compliance is not the whole answer

Passing a length or keyword requirement does not establish factual accuracy, relevance or sound reasoning. The authors identify instruction diversity and real-world applicability as areas to expand. Combine this signal with task-specific evaluation when selecting a model.

Instruction-Following Evaluation for Large Language Models · Metrics and limitations

Data source and credit: LLM Stats

Evidence available for this benchmark

This source record contains 67 numeric observations. The source distinguishes 67 model identities or configurations. A model may appear more than once; check the version, tools and run budget before comparing. Read results and row-level evidence →

Metric direction: higher is better

Start with these resources

Paper · task design and experiments · arxiv.org
Code · implementation and instructions · github.com
Catalog source · api.zeroeval.com

Cited documents and dates (1)

IFEval

Catalog collected on 2026-08-17; resources and fields reflect that saved source.

Recorded attention

Observed on: 2026-10-12

GitHub stars: 38,909
Hugging Face paper upvotes: 22

View source snapshot

Dated attention observations, not live counts or benchmark scores.

Evaluation context and evidence

Source: LLM Stats
Publisher: Google
Benchmark release: 2023-11-14
Collected: 2026-08-17
Modality: text

These are source-reported observations, not independent reruns by this site. Check the task version, evaluation protocol, unit and reporting date before comparing models. Results from different setups are not a universal ranking.

Reported results 67

Shown in source observation order, not ranked. Dates retain the precision provided by the source.

Model	Value	Unit	Protocol	Reported date and basis	Source
Qwen3.5-27B	0.95	Not recorded	Not recorded	2026-02-24 · model_announcement	LLM Stats
Qwen3.7-Plus	0.946	Not recorded	Not recorded	2026-05-31 · model_announcement	LLM Stats
Qwen3.7 Max	0.943	Not recorded	Not recorded	2026-05-19 · model_announcement	LLM Stats
Qwen3.6 Plus	0.943	Not recorded	Not recorded	2026-04-02 · model_announcement	LLM Stats
o3-mini	0.939	Not recorded	Not recorded	2025-01-30 · model_announcement	LLM Stats
Qwen3.5-122B-A10B	0.934	Not recorded	Not recorded	2026-02-24 · model_announcement	LLM Stats
Claude 3.7 Sonnet	0.932	Not recorded	Not recorded	2025-02-24 · model_announcement	LLM Stats
Qwen3.5-397B-A17B	0.926	Not recorded	Not recorded	2026-02-16 · model_announcement	LLM Stats
Llama 3.3 70B Instruct	0.921	Not recorded	Not recorded	2024-12-06 · model_announcement	LLM Stats
Nova Pro	0.921	Not recorded	Not recorded	2024-11-20 · model_announcement	LLM Stats
Qwen3.5-35B-A3B	0.919	Not recorded	Not recorded	2026-02-24 · model_announcement	LLM Stats
Qwen3.5-9B	0.915	Not recorded	Not recorded	2026-03-02 · model_announcement	LLM Stats
Gemma 3 27B	0.904	Not recorded	Not recorded	2025-03-12 · model_announcement	LLM Stats
Nemotron Nano 9B v2	0.903	Not recorded	Not recorded	2025-08-18 · model_announcement	LLM Stats
Gemma 3 4B	0.902	Not recorded	Not recorded	2025-03-12 · model_announcement	LLM Stats
Qwen3.5-4B	0.898	Not recorded	Not recorded	2026-03-02 · model_announcement	LLM Stats
Kimi K2-Instruct-0905	0.898	Not recorded	Not recorded	2025-09-05 · model_announcement	LLM Stats
Kimi K2 Instruct	0.898	Not recorded	Not recorded	2025-07-11 · model_announcement	LLM Stats
Nova Lite	0.897	Not recorded	Not recorded	2024-11-20 · model_announcement	LLM Stats
LongCat-Flash-Chat	0.8965	Not recorded	Not recorded	2025-08-29 · model_announcement	LLM Stats
Llama 3.1 Nemotron Ultra 253B v1	0.8945	Not recorded	Not recorded	2025-04-07 · model_announcement	LLM Stats
Qwen3-Next-80B-A3B-Thinking	0.889	Not recorded	Not recorded	2025-09-10 · model_announcement	LLM Stats
Gemma 3 12B	0.889	Not recorded	Not recorded	2025-03-12 · model_announcement	LLM Stats
Qwen3-235B-A22B-Instruct-2507	0.887	Not recorded	Not recorded	2025-07-22 · model_announcement	LLM Stats
Llama 3.1 405B Instruct	0.886	Not recorded	Not recorded	2024-07-23 · model_announcement	LLM Stats
Qwen3 VL 235B A22B Thinking	0.882	Not recorded	Not recorded	2025-09-22 · model_announcement	LLM Stats
GPT-4.5	0.882	Not recorded	Not recorded	2025-02-27 · model_announcement	LLM Stats
Qwen3 VL 235B A22B Instruct	0.878	Not recorded	Not recorded	2025-09-22 · model_announcement	LLM Stats
Qwen3 VL 32B Thinking	0.878	Not recorded	Not recorded	2025-09-22 · model_announcement	LLM Stats
Qwen3-235B-A22B-Thinking-2507	0.878	Not recorded	Not recorded	2025-07-25 · model_announcement	LLM Stats
Qwen3-Next-80B-A3B-Instruct	0.876	Not recorded	Not recorded	2025-09-10 · model_announcement	LLM Stats
Llama 3.1 70B Instruct	0.875	Not recorded	Not recorded	2024-07-23 · model_announcement	LLM Stats
GPT-4.1	0.874	Not recorded	Not recorded	2025-04-14 · model_announcement	LLM Stats
Kimi-k1.5	0.872	Not recorded	Not recorded	2025-01-20 · model_announcement	LLM Stats
Nova Micro	0.872	Not recorded	Not recorded	2024-11-20 · model_announcement	LLM Stats
DeepSeek-V3	0.861	Not recorded	Not recorded	2024-12-25 · model_announcement	LLM Stats
Qwen3 VL 30B A3B Instruct	0.858	Not recorded	Not recorded	2025-09-22 · model_announcement	LLM Stats
Phi 4 Reasoning Plus	0.849	Not recorded	Not recorded	2025-04-30 · model_announcement	LLM Stats
Sarvam-105B	0.848	Not recorded	Not recorded	2026-03-06 · model_announcement	LLM Stats
Qwen3 VL 32B Instruct	0.847	Not recorded	Not recorded	2025-09-22 · model_announcement	LLM Stats
GPT-4.1 mini	0.841	Not recorded	Not recorded	2025-04-14 · model_announcement	LLM Stats
Qwen2.5 72B Instruct	0.841	Not recorded	Not recorded	2024-09-19 · model_announcement	LLM Stats
QwQ-32B	0.839	Not recorded	Not recorded	2025-03-05 · model_announcement	LLM Stats
Qwen3 VL 8B Instruct	0.837	Not recorded	Not recorded	2025-09-22 · model_announcement	LLM Stats
Phi 4 Reasoning	0.834	Not recorded	Not recorded	2025-04-30 · model_announcement	LLM Stats
Qwen3 VL 8B Thinking	0.832	Not recorded	Not recorded	2025-09-22 · model_announcement	LLM Stats
Mistral Small 3 24B Instruct	0.829	Not recorded	Not recorded	2025-01-30 · model_announcement	LLM Stats
Qwen3 VL 4B Thinking	0.826	Not recorded	Not recorded	2025-09-22 · model_announcement	LLM Stats
LFM2.5-VL-3B	0.823	Not recorded	Not recorded	2026-08-12 · model_announcement	LLM Stats
Qwen3 VL 4B Instruct	0.823	Not recorded	Not recorded	2025-09-22 · model_announcement	LLM Stats
Qwen3 VL 30B A3B Thinking	0.817	Not recorded	Not recorded	2025-09-22 · model_announcement	LLM Stats
GPT-4o	0.81	Not recorded	Not recorded	2024-08-06 · model_announcement	LLM Stats
Llama 3.1 8B Instruct	0.804	Not recorded	Not recorded	2024-07-23 · model_announcement	LLM Stats
Gemma 3 1B	0.802	Not recorded	Not recorded	2025-03-12 · model_announcement	LLM Stats
Llama 3.1 Nemotron Nano 8B V1	0.793	Not recorded	Not recorded	2025-03-18 · model_announcement	LLM Stats
Qwen3.5-2B	0.786	Not recorded	Not recorded	2026-03-02 · model_announcement	LLM Stats
Llama 3.2 3B Instruct	0.774	Not recorded	Not recorded	2024-09-25 · model_announcement	LLM Stats
MiniCPM-SALA	0.7634	Not recorded	Not recorded	2026-02-11 · model_announcement	LLM Stats
North Micro Vision Instruct	0.749	Not recorded	Not recorded	2026-08-12 · model_announcement	LLM Stats
Granite 3.3 8B Base	0.7482	Not recorded	Not recorded	2025-04-16 · model_announcement	LLM Stats
Granite 3.3 8B Instruct	0.7482	Not recorded	Not recorded	2025-04-16 · model_announcement	LLM Stats
GPT-4.1 nano	0.745	Not recorded	Not recorded	2025-04-14 · model_announcement	LLM Stats
Qwen2.5 7B Instruct	0.712	Not recorded	Not recorded	2024-09-19 · model_announcement	LLM Stats
IBM Granite 4.0 Tiny Preview	0.63	Not recorded	Not recorded	2025-05-02 · model_announcement	LLM Stats
Phi 4	0.63	Not recorded	Not recorded	2024-12-12 · model_announcement	LLM Stats
Pixtral-12B	0.613	Not recorded	Not recorded	2024-09-17 · model_announcement	LLM Stats
Qwen3.5-0.8B	0.44	Not recorded	Not recorded	2026-03-02 · model_announcement	LLM Stats

Related records in other sources

IFEval

Constraints that a program can check

Checkers replace a subjective judge

Prompt-level versus instruction-level

Strict and loose are different checks

Compliance is not the whole answer

Evidence available for this benchmark

Start with these resources

Recorded attention

Evaluation context and evidence

Reported results 67

Explore related benchmarks

Put this benchmark in context

Six Navers Lab / Einsia benchmarks and the work they test

AI benchmarks and leaderboards are different things

How to compare AI benchmark scores without mixing protocols

Four new agent benchmarks: adaptation, safety and video

Continue exploring

Related records in other sources