CompoSET

What the test isolates

Each test uses two near-identical images and two corresponding captions. The edit changes one named factor while keeping the scene fixed; ranking four image-caption combinations tests compositional binding rather than recognition of a different scene.

CompoSET paper v1 — task, tables and limitations · CompoSET author evaluation repository

Dataset and construction

The release has 1,776 base/foil instances across 80 synthetic scenes, 16 edit types and three caption lengths. Edits cover identity/appearance, quantity, spatial relations and activity/state. Claude Sonnet 4.6 supplies scene/caption specifications and Gemini 2.5 Flash Image supplies imagery; an author manually reviews candidates. There is no training split.

CompoSET paper v1 — task, tables and limitations · CompoSET official dataset

Read Group accuracy correctly

Image-to-text and text-to-image each require both correct pairings to outrank their alternatives. Group succeeds only when both directions pass; there is no partial credit. The page records the published Group percentages. The exchangeable-score chance reference is 16.67%, not the product of two independent direction probabilities.

CompoSET paper v1 — task, tables and limitations

Separate the reported experiments

Table 8 covers all 22 dual encoders with deterministic cosine scoring. Table 9 reports 14 models separately by edit type. Tables 11–12 use a stratified 200-instance generative slice: caption likelihood for open-weight models, or temperature-zero pairwise selection with option-order counterbalancing for API models. Their subsets and scoring primitives differ.

CompoSET paper v1 — task, tables and limitations · CompoSET author evaluation repository

What the results cannot establish

This is a single-edit, static-image, English retrieval test. Synthetic imagery, one reviewing author and only 80 underlying scenes limit generalization; role swaps have just 45 instances. Scene clustering widens uncertainty intervals. A large score on one edit subset is not the full benchmark score, and the small generative slice is not a head-to-head ranking against the dual-encoder pool.

CompoSET paper v1 — task, tables and limitations

Find the original release

The paper links the author evaluation repository and Hugging Face dataset. The repository exports per-pair I2T, T2I and Group values and documents checkpoint requirements. These are publisher-reported results; this page has not rerun the models.

CompoSET paper v1 — task, tables and limitations · CompoSET author evaluation repository · CompoSET official dataset

Full benchmark: 22 dual encoders

Table 8 reports overall strict Group percentages; these are the only CompoSET rows in the normalized primary series.

Full dataset, deterministic cosine scoring. These table intervals are row-level; scene-clustered intervals in Appendix C are wider.

CompoSET paper v1 — task, tables and limitations

Edit subsets: Group by model and edit type

Table 9 reports 224 subset measurements. Each row conditions on one edit type; none is an overall benchmark score.

Fourteen dual encoders; short/medium/long captions pooled. Preserve the printed point estimate and interval. Role swaps have only 45 base pairs (135 pair-caption observations).

CompoSET paper v1 — task, tables and limitations

Edit and caption-length summary across 14 models

Table 10 averages Group scores across the fourteen models. These aggregate values are not individual model results.

Model-averaged subset diagnostics from Table 10; excluded from the primary score series.

CompoSET paper v1 — task, tables and limitations

200-instance generative slice

Table 11 uses a different scoring primitive from dual encoders. Open-weight models score caption likelihood; API models choose between options with order counterbalancing.

Stratified 200-instance subset. API temperature 0; log-likelihood is length-normalized. I2T is the paper’s preferred diagnostic for the likelihood slice; Group is shown as reported, not mixed into overall results.

CompoSET paper v1 — task, tables and limitations

API models by caption length

Table 12 reports Group for each verbosity tier in the small API-model slice.

Approximately 67 instances per tier; temperature 0 and order counterbalancing. These values condition on a subset and are not full-benchmark scores.

CompoSET paper v1 — task, tables and limitations