How to compare AI benchmark scores without mixing protocols
A practical sequence for checking task versions, metrics, tools, budgets, uncertainty and reporting sources.
Two scores become a useful comparison only after you establish what was tested and how. Check task identity first, then the metric, system configuration, resources and uncertainty.
First match the test
Begin with the exact benchmark, version and subset. Similar names do not guarantee identical questions, environments or grading. SWE-bench maintains distinct task sets and leaderboard settings; a result from one should not be inserted into another's order without an explicit mapping. The same caution applies when a benchmark repairs its verifier or removes problematic examples.
Write down the evaluation split and any exclusions. If a report omits failed environment setups, uses only supported languages or selects a small task sample, its denominator may differ from another report's. A matching title cannot resolve that difference. When these details are unavailable, keep both observations visible but label the comparison as unresolved.
Read the metric before the number
Identify the measured outcome, unit and direction. Accuracy, task completion, a judge's preference and attack success are not synonyms. A higher value can indicate better task performance or a more successful attack against a defense. A proportion and a percentage may describe the same quantity, but conversion is justified only when the source defines the scale.
Next identify the attempt rule. A first-attempt result differs from success across several generated candidates. HumanEval's pass-at-k formulation makes the sampling opportunity part of the measurement. More attempts can be relevant to your application, but they answer a different operational question. Preserve the reported metric rather than turning every number into an unlabeled percentage.
HumanEval and pass-at-k evaluation · HarmBench evaluation pipeline
Compare complete systems under stated conditions
Record the model version and the software around it. Search, code execution, retrieved documents, interface metadata and external memory can all provide information or actions unavailable to another run. For computer use, screenshots and structured interface information are materially different observation channels. For agents, the surrounding control loop may decide how failures are retried.
Compare time, token and interaction budgets as well as the final result. An unlimited run is not the same product experience as a tightly bounded one. If your use case has a latency ceiling or a fixed spend, evaluate the relevant operating point. Do not infer that a higher completion score is a free improvement.
OSWorld observation and action settings · Berkeley Function Calling Leaderboard evaluation settings
Separate an observed gap from a reliable conclusion
Inspect the task count, repeated runs and any uncertainty intervals reported by the evaluator. A small numerical difference is not automatically meaningful, especially when the system samples different answers on different attempts. If uncertainty is not reported, say that it is unavailable; do not invent an interval or a significance claim.
Look for changes in evaluation design that could explain a gap. Long-context comparisons, for example, depend on the actual input length, task construction and truncation policy, not just the model's advertised window. Error examples can show whether a difference affects the work you care about. An aggregate alone cannot reveal that pattern.
Keep a comparison note you can audit
Save the source URL, reporting date, exact metric label and relevant protocol description with each observation. Distinguish the report's publication date from an experiment date or benchmark release date. Also distinguish an evaluator's direct measurement from a model developer repeating someone else's result. Those relationships help you check a claim without treating repeated reports as independent confirmation.
In a directory containing multiple sources, start from the source record and use a documented comparison group when one exists. Do not merge records by model name or benchmark title alone. A zero is a measurement; a missing value is not zero. If the available evidence cannot support a ranking, a table of separately attributed observations is still useful.
Finish with a narrow conclusion: which system performed better on which task, under which conditions, with what remaining uncertainty. Then run a small evaluation on representative work from your own application. Public results help select candidates and identify risks; they do not remove the need to test the decision you are about to make.