AI EVALUATION, IN CONTEXT.BY EQUALITY MACHINE
Benchmark Radar.Benchmarks. Model results. Original sources.
/

Model reports

AI4AI-Bench

Can a coding agent improve the algorithm inside an existing AI research project? AI4AI-Bench lets it rewrite a working repository, then reruns only the submitted source under a fixed training and evaluation budget.

Official projectOriginal paperOfficial repositoryCatalog source

AI Benchmark Radar / Equality Machine · Reviewed 2026-10-27

The ten-task AI4AI-Bench suite, with paper v1 results and the separately identified v1.5 official chart data. Model, harness and reasoning effort define each evaluated configuration.

Ten different algorithm families

The ten frozen projects cover supervised fine-tuning, agentic reinforcement learning, on-policy distillation, reward modeling, preference optimization, diffusion reinforcement learning, unlearning, graph diffusion, model merging and pruning. The model and assets are fixed; the agent may change the learning method and training code. Model Soup and OWL do not train: they combine existing checkpoints or prune a model.

AI4AI-Bench paper v1 — task protocol, scoring and Tables 2–3 · Official AI4AI-Bench repository

Exploration and final execution are separate

Each configuration gets four hours on one NVIDIA B300 to inspect code and run experiments using a proxy metric. Only source code crosses into a fresh formal environment. Eight training tasks then run from initialization for up to twelve hours, with the best result among at most the three latest checkpoints. The two non-training tasks execute once; twelve hours is their verification ceiling.

AI4AI-Bench paper v1 — task protocol, scoring and Tables 2–3

Read the normalized score correctly

Different raw task metrics are mapped to a common 0–1 scale before their mean is taken. Zero is an uninformative model, 0.1 is the unchanged repository recipe under the same conditions, and 1 is the metric-defined optimum. Perplexity uses a logarithmic coordinate. A score of 0.288 is not 28.8% of tasks solved. The paper assigns zero to submissions that return no model; raw execution receipts with missing or invalid results must not be converted into a task-metric zero.

AI4AI-Bench paper v1 — task protocol, scoring and Tables 2–3 · Official evaluation and replay documentation

Compare configurations, not pooled model names

The study contains 29 configurations of six systems, each attempting all ten tasks. The paper reports Claude Opus 5 at medium effort at 0.288, while 0.250 averages its five effort settings. These answer different questions. The official JSON labels itself v1.5 and preserves more decimals, such as 0.2881 for medium effort; retain the cited source and printed precision instead of replacing one with the other.

AI4AI-Bench paper v1 — task protocol, scoring and Tables 2–3 · Official chart data — AI4AI-Bench v1.5 mapped scores

What this can and cannot establish

This measures algorithm changes in ten selected projects under one GPU and fixed time budgets, not a demonstrated recursive self-improvement cycle. The baseline is the original repository recipe, not a human expert. Some cheap proxies share a corpus with final evaluation, so separation does not guarantee disjoint samples. The public release supports self-hosted evaluation; its documentation says an official blind service is not yet available and local warning-mode runs are non-official.

AI4AI-Bench paper v1 — task protocol, scoring and Tables 2–3 · Official evaluation and replay documentation

All 29 official chart configurations

These are the exact mean values from the official points.json labeled “AI4AI-Bench v1.5 mapped scores”, observed on 27 October 2026. Each configuration attempts all ten tasks. The source provides no exact publication date for this data file, so the observation date is not presented as a new release or score-report date.

Official v1.5 chart data · all ten tasks · observed 27 October 2026
ModelOrganizationHarnessReasoning effortMean normalized score
Claude Opus 5AnthropicClaude Codelow0.2722
Claude Opus 5AnthropicClaude Codemedium0.2881
Claude Opus 5AnthropicClaude Codehigh0.2776
Claude Opus 5AnthropicClaude Codexhigh0.1898
Claude Opus 5AnthropicClaude Codemax0.2245
Claude Sonnet 5AnthropicClaude Codelow0.0991
Claude Sonnet 5AnthropicClaude Codemedium0.1092
Claude Sonnet 5AnthropicClaude Codehigh0.216
Claude Sonnet 5AnthropicClaude Codexhigh0.137
Claude Sonnet 5AnthropicClaude Codemax0.1635
Kimi K3Moonshot AIClaude Codemax0.1738
GPT-5.6 SolOpenAICodexnone0.1261
GPT-5.6 SolOpenAICodexlow0.1976
GPT-5.6 SolOpenAICodexmedium0.144
GPT-5.6 SolOpenAICodexhigh0.235
GPT-5.6 SolOpenAICodexxhigh0.1952
GPT-5.6 SolOpenAICodexmax0.2455
GPT-5.6 TerraOpenAICodexnone0.065
GPT-5.6 TerraOpenAICodexlow0.1085
GPT-5.6 TerraOpenAICodexmedium0.1349
GPT-5.6 TerraOpenAICodexhigh0.1293
GPT-5.6 TerraOpenAICodexxhigh0.1438
GPT-5.6 TerraOpenAICodexmax0.2282
GPT-5.6 LunaOpenAICodexnone0.0914
GPT-5.6 LunaOpenAICodexlow0.0786
GPT-5.6 LunaOpenAICodexmedium0.1242
GPT-5.6 LunaOpenAICodexhigh0.1191
GPT-5.6 LunaOpenAICodexxhigh0.1468
GPT-5.6 LunaOpenAICodexmax0.1393

Original JSON precision retained. The scale is 0–1 with the unchanged recipe at 0.1; these are not percentages. Scores evaluate model, harness and effort together. Paper Table 3 reports separate, lower-precision values; this table does not overwrite them.

Official chart data — AI4AI-Bench v1.5 mapped scores · AI4AI-Bench official project and results · AI4AI-Bench paper v1 — task protocol, scoring and Tables 2–3

Six system averages across their tested effort settings

The official file also publishes these six pooled system means. They combine the configurations listed in each row and therefore have different denominators: Claude has five tested effort settings, GPT-5.6 has six, and Kimi K3 has one. They are not additional single-effort configurations.

Official pooled means · configuration-task cells, not unique task counts
ModelHarnessIncluded effort settingsConfiguration-task cellsPooled mean score
Claude Opus 5Claude Codelow, medium, high, xhigh, max500.25
GPT-5.6 SolCodexnone, low, medium, high, xhigh, max600.191
Kimi K3Claude Codemax100.174
Claude Sonnet 5Claude Codelow, medium, high, xhigh, max500.145
GPT-5.6 TerraCodexnone, low, medium, high, xhigh, max600.135
GPT-5.6 LunaCodexnone, low, medium, high, xhigh, max600.117

All rows use the same ten-task suite, but pool different numbers of configurations. These author-reported averages are retained exactly; they must not replace the 29 individual configuration scores.

Official chart data — AI4AI-Bench v1.5 mapped scores · AI4AI-Bench paper v1 — task protocol, scoring and Tables 2–3

Original publication

The ten-task AI4AI-Bench suite, with paper v1 results and the separately identified v1.5 official chart data. Model, harness and reasoning effort define each evaluated configuration.

Reviewed 2026-10-27

Evidence available for this benchmark

This source record contains 29 numeric observations. The source distinguishes 6 model identities or configurations. A model may appear more than once; check the version, tools and run budget before comparing. Read results and row-level evidence →

Reported metric
Mean normalized score
Recorded unit
points
Metric direction
higher is better

Start with these resources

Cited documents and dates (1)

Recorded attention

Observed on: 2026-10-11

Hugging Face paper upvotes
3

View source snapshot

Dated attention observations, not live counts or benchmark scores.

Evaluation context and evidence

Source
Model reports
Publisher
Not recorded
Benchmark release
2026-08-20
Collected
Not recorded
Modality
Not recorded

These are source-reported observations, not independent reruns by this site. Check the task version, evaluation protocol, unit and reporting date before comparing models. Results from different setups are not a universal ranking.

Reported results 29

Shown in source observation order, not ranked. Dates retain the precision provided by the source.

ModelValueUnitProtocolReported date and basisSource
Claude Opus 50.278pointsClaude Code; reasoning effort high; 10 tasks; 4 h exploration on 1 NVIDIA B300; 8 training tasks: clean-start run up to 12 h, best of up to 3 latest checkpoints; 2 non-training tasks: one execution under 12 h verification ceiling; mean mapped score (paper v1 Table 3)2026-08-20 · document_publicationAI4AI-Bench paper v1 — mean mapped scores, Table 3
Claude Sonnet 50.216pointsClaude Code; reasoning effort high; 10 tasks; 4 h exploration on 1 NVIDIA B300; 8 training tasks: clean-start run up to 12 h, best of up to 3 latest checkpoints; 2 non-training tasks: one execution under 12 h verification ceiling; mean mapped score (paper v1 Table 3)2026-08-20 · document_publicationAI4AI-Bench paper v1 — mean mapped scores, Table 3
Claude Opus 50.272pointsClaude Code; reasoning effort low; 10 tasks; 4 h exploration on 1 NVIDIA B300; 8 training tasks: clean-start run up to 12 h, best of up to 3 latest checkpoints; 2 non-training tasks: one execution under 12 h verification ceiling; mean mapped score (paper v1 Table 3)2026-08-20 · document_publicationAI4AI-Bench paper v1 — mean mapped scores, Table 3
Claude Sonnet 50.099pointsClaude Code; reasoning effort low; 10 tasks; 4 h exploration on 1 NVIDIA B300; 8 training tasks: clean-start run up to 12 h, best of up to 3 latest checkpoints; 2 non-training tasks: one execution under 12 h verification ceiling; mean mapped score (paper v1 Table 3)2026-08-20 · document_publicationAI4AI-Bench paper v1 — mean mapped scores, Table 3
Claude Opus 50.225pointsClaude Code; reasoning effort max; 10 tasks; 4 h exploration on 1 NVIDIA B300; 8 training tasks: clean-start run up to 12 h, best of up to 3 latest checkpoints; 2 non-training tasks: one execution under 12 h verification ceiling; mean mapped score (paper v1 Table 3)2026-08-20 · document_publicationAI4AI-Bench paper v1 — mean mapped scores, Table 3
Claude Sonnet 50.163pointsClaude Code; reasoning effort max; 10 tasks; 4 h exploration on 1 NVIDIA B300; 8 training tasks: clean-start run up to 12 h, best of up to 3 latest checkpoints; 2 non-training tasks: one execution under 12 h verification ceiling; mean mapped score (paper v1 Table 3)2026-08-20 · document_publicationAI4AI-Bench paper v1 — mean mapped scores, Table 3
Kimi K30.174pointsClaude Code; reasoning effort max; 10 tasks; 4 h exploration on 1 NVIDIA B300; 8 training tasks: clean-start run up to 12 h, best of up to 3 latest checkpoints; 2 non-training tasks: one execution under 12 h verification ceiling; mean mapped score (paper v1 Table 3)2026-08-20 · document_publicationAI4AI-Bench paper v1 — mean mapped scores, Table 3
Claude Opus 50.288pointsClaude Code; reasoning effort medium; 10 tasks; 4 h exploration on 1 NVIDIA B300; 8 training tasks: clean-start run up to 12 h, best of up to 3 latest checkpoints; 2 non-training tasks: one execution under 12 h verification ceiling; mean mapped score (paper v1 Table 3)2026-08-20 · document_publicationAI4AI-Bench paper v1 — mean mapped scores, Table 3
Claude Sonnet 50.109pointsClaude Code; reasoning effort medium; 10 tasks; 4 h exploration on 1 NVIDIA B300; 8 training tasks: clean-start run up to 12 h, best of up to 3 latest checkpoints; 2 non-training tasks: one execution under 12 h verification ceiling; mean mapped score (paper v1 Table 3)2026-08-20 · document_publicationAI4AI-Bench paper v1 — mean mapped scores, Table 3
Claude Opus 50.19pointsClaude Code; reasoning effort xhigh; 10 tasks; 4 h exploration on 1 NVIDIA B300; 8 training tasks: clean-start run up to 12 h, best of up to 3 latest checkpoints; 2 non-training tasks: one execution under 12 h verification ceiling; mean mapped score (paper v1 Table 3)2026-08-20 · document_publicationAI4AI-Bench paper v1 — mean mapped scores, Table 3
Claude Sonnet 50.137pointsClaude Code; reasoning effort xhigh; 10 tasks; 4 h exploration on 1 NVIDIA B300; 8 training tasks: clean-start run up to 12 h, best of up to 3 latest checkpoints; 2 non-training tasks: one execution under 12 h verification ceiling; mean mapped score (paper v1 Table 3)2026-08-20 · document_publicationAI4AI-Bench paper v1 — mean mapped scores, Table 3
GPT-5.6 Luna0.119pointsCodex; reasoning effort high; 10 tasks; 4 h exploration on 1 NVIDIA B300; 8 training tasks: clean-start run up to 12 h, best of up to 3 latest checkpoints; 2 non-training tasks: one execution under 12 h verification ceiling; mean mapped score (paper v1 Table 3)2026-08-20 · document_publicationAI4AI-Bench paper v1 — mean mapped scores, Table 3
GPT-5.6 Sol0.235pointsCodex; reasoning effort high; 10 tasks; 4 h exploration on 1 NVIDIA B300; 8 training tasks: clean-start run up to 12 h, best of up to 3 latest checkpoints; 2 non-training tasks: one execution under 12 h verification ceiling; mean mapped score (paper v1 Table 3)2026-08-20 · document_publicationAI4AI-Bench paper v1 — mean mapped scores, Table 3
GPT-5.6 Terra0.129pointsCodex; reasoning effort high; 10 tasks; 4 h exploration on 1 NVIDIA B300; 8 training tasks: clean-start run up to 12 h, best of up to 3 latest checkpoints; 2 non-training tasks: one execution under 12 h verification ceiling; mean mapped score (paper v1 Table 3)2026-08-20 · document_publicationAI4AI-Bench paper v1 — mean mapped scores, Table 3
GPT-5.6 Luna0.079pointsCodex; reasoning effort low; 10 tasks; 4 h exploration on 1 NVIDIA B300; 8 training tasks: clean-start run up to 12 h, best of up to 3 latest checkpoints; 2 non-training tasks: one execution under 12 h verification ceiling; mean mapped score (paper v1 Table 3)2026-08-20 · document_publicationAI4AI-Bench paper v1 — mean mapped scores, Table 3
GPT-5.6 Sol0.198pointsCodex; reasoning effort low; 10 tasks; 4 h exploration on 1 NVIDIA B300; 8 training tasks: clean-start run up to 12 h, best of up to 3 latest checkpoints; 2 non-training tasks: one execution under 12 h verification ceiling; mean mapped score (paper v1 Table 3)2026-08-20 · document_publicationAI4AI-Bench paper v1 — mean mapped scores, Table 3
GPT-5.6 Terra0.109pointsCodex; reasoning effort low; 10 tasks; 4 h exploration on 1 NVIDIA B300; 8 training tasks: clean-start run up to 12 h, best of up to 3 latest checkpoints; 2 non-training tasks: one execution under 12 h verification ceiling; mean mapped score (paper v1 Table 3)2026-08-20 · document_publicationAI4AI-Bench paper v1 — mean mapped scores, Table 3
GPT-5.6 Luna0.139pointsCodex; reasoning effort max; 10 tasks; 4 h exploration on 1 NVIDIA B300; 8 training tasks: clean-start run up to 12 h, best of up to 3 latest checkpoints; 2 non-training tasks: one execution under 12 h verification ceiling; mean mapped score (paper v1 Table 3)2026-08-20 · document_publicationAI4AI-Bench paper v1 — mean mapped scores, Table 3
GPT-5.6 Sol0.245pointsCodex; reasoning effort max; 10 tasks; 4 h exploration on 1 NVIDIA B300; 8 training tasks: clean-start run up to 12 h, best of up to 3 latest checkpoints; 2 non-training tasks: one execution under 12 h verification ceiling; mean mapped score (paper v1 Table 3)2026-08-20 · document_publicationAI4AI-Bench paper v1 — mean mapped scores, Table 3
GPT-5.6 Terra0.228pointsCodex; reasoning effort max; 10 tasks; 4 h exploration on 1 NVIDIA B300; 8 training tasks: clean-start run up to 12 h, best of up to 3 latest checkpoints; 2 non-training tasks: one execution under 12 h verification ceiling; mean mapped score (paper v1 Table 3)2026-08-20 · document_publicationAI4AI-Bench paper v1 — mean mapped scores, Table 3
GPT-5.6 Luna0.124pointsCodex; reasoning effort medium; 10 tasks; 4 h exploration on 1 NVIDIA B300; 8 training tasks: clean-start run up to 12 h, best of up to 3 latest checkpoints; 2 non-training tasks: one execution under 12 h verification ceiling; mean mapped score (paper v1 Table 3)2026-08-20 · document_publicationAI4AI-Bench paper v1 — mean mapped scores, Table 3
GPT-5.6 Sol0.144pointsCodex; reasoning effort medium; 10 tasks; 4 h exploration on 1 NVIDIA B300; 8 training tasks: clean-start run up to 12 h, best of up to 3 latest checkpoints; 2 non-training tasks: one execution under 12 h verification ceiling; mean mapped score (paper v1 Table 3)2026-08-20 · document_publicationAI4AI-Bench paper v1 — mean mapped scores, Table 3
GPT-5.6 Terra0.135pointsCodex; reasoning effort medium; 10 tasks; 4 h exploration on 1 NVIDIA B300; 8 training tasks: clean-start run up to 12 h, best of up to 3 latest checkpoints; 2 non-training tasks: one execution under 12 h verification ceiling; mean mapped score (paper v1 Table 3)2026-08-20 · document_publicationAI4AI-Bench paper v1 — mean mapped scores, Table 3
GPT-5.6 Luna0.091pointsCodex; reasoning effort none; 10 tasks; 4 h exploration on 1 NVIDIA B300; 8 training tasks: clean-start run up to 12 h, best of up to 3 latest checkpoints; 2 non-training tasks: one execution under 12 h verification ceiling; mean mapped score (paper v1 Table 3)2026-08-20 · document_publicationAI4AI-Bench paper v1 — mean mapped scores, Table 3
GPT-5.6 Sol0.126pointsCodex; reasoning effort none; 10 tasks; 4 h exploration on 1 NVIDIA B300; 8 training tasks: clean-start run up to 12 h, best of up to 3 latest checkpoints; 2 non-training tasks: one execution under 12 h verification ceiling; mean mapped score (paper v1 Table 3)2026-08-20 · document_publicationAI4AI-Bench paper v1 — mean mapped scores, Table 3
GPT-5.6 Terra0.065pointsCodex; reasoning effort none; 10 tasks; 4 h exploration on 1 NVIDIA B300; 8 training tasks: clean-start run up to 12 h, best of up to 3 latest checkpoints; 2 non-training tasks: one execution under 12 h verification ceiling; mean mapped score (paper v1 Table 3)2026-08-20 · document_publicationAI4AI-Bench paper v1 — mean mapped scores, Table 3
GPT-5.6 Luna0.147pointsCodex; reasoning effort xhigh; 10 tasks; 4 h exploration on 1 NVIDIA B300; 8 training tasks: clean-start run up to 12 h, best of up to 3 latest checkpoints; 2 non-training tasks: one execution under 12 h verification ceiling; mean mapped score (paper v1 Table 3)2026-08-20 · document_publicationAI4AI-Bench paper v1 — mean mapped scores, Table 3
GPT-5.6 Sol0.195pointsCodex; reasoning effort xhigh; 10 tasks; 4 h exploration on 1 NVIDIA B300; 8 training tasks: clean-start run up to 12 h, best of up to 3 latest checkpoints; 2 non-training tasks: one execution under 12 h verification ceiling; mean mapped score (paper v1 Table 3)2026-08-20 · document_publicationAI4AI-Bench paper v1 — mean mapped scores, Table 3
GPT-5.6 Terra0.144pointsCodex; reasoning effort xhigh; 10 tasks; 4 h exploration on 1 NVIDIA B300; 8 training tasks: clean-start run up to 12 h, best of up to 3 latest checkpoints; 2 non-training tasks: one execution under 12 h verification ceiling; mean mapped score (paper v1 Table 3)2026-08-20 · document_publicationAI4AI-Bench paper v1 — mean mapped scores, Table 3

Explore related benchmarks

Suggestions based on shared evidence or source categories, not verified equivalents.

PresentBenchShared source categories →Kimi Code Bench v2Shared source categories →SpreadsheetBench 2Shared source categories →CoWorkBenchShared source categories →OSWorld-VerifiedShared source categories →BrowseCompShared source categories →LiveSQLBenchShared source categories →Siren AgentDojo UtilityShared source categories →

Put this benchmark in context

Coding benchmarksTools and agents benchmarks

Continue exploring

Search the catalog using this record’s source labels. Search matches are not verified equivalents.

agents →Model reports · all records →