AI EVALUATION, IN CONTEXT.BY EQUALITY MACHINE
Benchmark Radar.Benchmarks. Model results. Original sources.
/

Multimodal benchmarks

Find out which combinations of images, text, audio or video a benchmark actually evaluates.

Multimodal evaluation concerns tasks that use more than one kind of information. Reading a chart requires identifying visual marks and relating them to the question; understanding a video can also require tracking events over time. The label alone does not tell you which modalities are present. MMMU, for example, focuses on questions combining text and images across academic subjects. It should not stand in for speech or video evaluation.

What these benchmarks test

  • Perception tasks ask what is visible or audible; reasoning tasks ask what follows from that evidence.
  • Document and diagram tasks may depend on layout, small text, spatial relationships or specialized notation.
  • Interactive tasks add actions and state changes to the interpretation of visual input.

How to read the results

  • Check input modalities and preprocessing, including image resolution, cropping, OCR and sampled video frames.
  • Separate perception errors from reasoning errors where the benchmark exposes those categories.
  • Match answer format and grader: multiple-choice accuracy, free-text judging and action success measure different outcomes.

What to watch for

  • An aggregate can hide weak performance on a particular modality or subject.
  • Access to extracted text or interface metadata can change the task substantially; do not assume every run uses pixels alone.

References

  1. MMMU official project
  2. OSWorld input settings and task environment

Benchmarks in this topic 372 source records

Search and filter this topic

Related topics

Computer use benchmarksReasoning and math benchmarksScience benchmarks