AI EVALUATION, IN CONTEXT.BY EQUALITY MACHINE
Benchmark Radar.Benchmarks. Model results. Original sources.
/

Long context benchmarks

Distinguish accepting a long input from reliably retrieving, combining and summarizing information inside it.

Long context benchmarks test the use of information spread across a large input. A model's advertised context window describes capacity, not how well it uses that space. Finding a hidden phrase is one useful check, but document comparison and narrative summarization require additional abilities. RULER includes tasks beyond simple retrieval; HELM Long Context combines several task families instead of treating input length as a quality score.

What these benchmarks test

  • Retrieval checks whether the relevant evidence can be located among distracting material.
  • Multi-hop and aggregation tasks require combining facts from different positions.
  • Summarization tests coverage and fidelity when the output must compress a much larger source.

How to read the results

  • Record tested input length, tokenizer, truncation policy and the position of relevant information.
  • Examine performance by length and task rather than relying only on an overall average.
  • Identify whether the full text reaches the model or an external retrieval system selects passages first.

What to watch for

  • Synthetic distractors can be easier to ignore than contradictory evidence in real documents.
  • A long sequence of agent actions is not automatically a long context test; memory may be summarized or discarded between steps.

References

  1. RULER benchmark and configurable tasks
  2. HELM Long Context methodology

Benchmarks in this topic 81 source records

Search and filter this topic

Related topics

Language benchmarksReasoning and math benchmarksTools and agents benchmarks