CompoSET — Model reports · AI Benchmark Radar

AI Benchmark Radar / Equality Machine · Reviewed 2026-11-02

Author-reported results in arXiv v1 (30 October 2026), with the paper’s dataset freeze 662230d. Full retrieval scores, edit subsets and generative slices remain separate.

What the test isolates

Each test uses two near-identical images and two corresponding captions. The edit changes one named factor while keeping the scene fixed; ranking four image-caption combinations tests compositional binding rather than recognition of a different scene.

CompoSET paper v1 — task, tables and limitations · CompoSET author evaluation repository

Dataset and construction

The release has 1,776 base/foil instances across 80 synthetic scenes, 16 edit types and three caption lengths. Edits cover identity/appearance, quantity, spatial relations and activity/state. Claude Sonnet 4.6 supplies scene/caption specifications and Gemini 2.5 Flash Image supplies imagery; an author manually reviews candidates. There is no training split.

CompoSET paper v1 — task, tables and limitations · CompoSET official dataset

Read Group accuracy correctly

Image-to-text and text-to-image each require both correct pairings to outrank their alternatives. Group succeeds only when both directions pass; there is no partial credit. The page records the published Group percentages. The exchangeable-score chance reference is 16.67%, not the product of two independent direction probabilities.

CompoSET paper v1 — task, tables and limitations

Separate the reported experiments

Table 8 covers all 22 dual encoders with deterministic cosine scoring. Table 9 reports 14 models separately by edit type. Tables 11–12 use a stratified 200-instance generative slice: caption likelihood for open-weight models, or temperature-zero pairwise selection with option-order counterbalancing for API models. Their subsets and scoring primitives differ.

CompoSET paper v1 — task, tables and limitations · CompoSET author evaluation repository

What the results cannot establish

This is a single-edit, static-image, English retrieval test. Synthetic imagery, one reviewing author and only 80 underlying scenes limit generalization; role swaps have just 45 instances. Scene clustering widens uncertainty intervals. A large score on one edit subset is not the full benchmark score, and the small generative slice is not a head-to-head ranking against the dual-encoder pool.

CompoSET paper v1 — task, tables and limitations

Find the original release

The paper links the author evaluation repository and Hugging Face dataset. The repository exports per-pair I2T, T2I and Group values and documents checkpoint requirements. These are publisher-reported results; this page has not rerun the models.

CompoSET paper v1 — task, tables and limitations · CompoSET author evaluation repository · CompoSET official dataset

Full benchmark: 22 dual encoders

Table 8 reports overall strict Group percentages; these are the only CompoSET rows in the normalized primary series.

Full benchmark: 22 dual encoders
Model	Overall Group (%)	Published 95% CI
CLIP B/32	4.4	[3.9,5.0]
CLIP B/16 (OpenAI)	4.8	[4.2,5.4]
CLIP B/16 (LAION-400M)	8.4	[7.6,9.1]
CLIP B/16 (LAION-2B)	7.8	[7.1,8.5]
CLIP L/14 (OpenAI)	5.0	[4.5,5.7]
CLIP L/14 (LAION-2B)	8.4	[7.7,9.2]
CLIP H/14 (LAION-2B)	8.4	[7.7,9.1]
SigLIP B/16	10.2	[9.4,11.0]
SigLIP L/16-384	12.7	[11.9,13.7]
SigLIP SO400M/14	13.5	[12.6,14.4]
SigLIP-2 B/16	11.3	[10.5,12.2]
SigLIP-2 L/16-384	13.6	[12.7,14.6]
SigLIP-2 SO400M/14	13.9	[13.0,14.9]
NegCLIP B/32	7.0	[6.4,7.7]
CLoVe B/32	5.5	[5.0,6.2]
DreamLIP B/16 (30M)	13.7	[12.8,14.6]
DFN B/16	8.4	[7.7,9.2]
DFN H/14	10.8	[10.0,11.7]
EVA-02 B/16	7.8	[7.1,8.5]
EVA-02 L/14	8.6	[7.9,9.4]
MetaCLIP-2 H/14	8.9	[8.2,9.7]
Jina-v4	8.5	[7.8,9.3]

Full dataset, deterministic cosine scoring. These table intervals are row-level; scene-clustered intervals in Appendix C are wider.

CompoSET paper v1 — task, tables and limitations

Edit subsets: Group by model and edit type

Table 9 reports 224 subset measurements. Each row conditions on one edit type; none is an overall benchmark score.

Edit subsets: Group by model and edit type
Model	Edit type	Pooled n	Group % [95% CI]
CLIP B/32	modify_color	618	20.4 [17.2,23.6]
CLIP B/16 LAION-2B	modify_color	618	29.9 [26.3,33.5]
CLIP H/14 LAION-2B	modify_color	618	28.8 [25.2,32.4]
SigLIP B/16	modify_color	618	37.2 [33.4,41.0]
SigLIP SO400M	modify_color	618	39.5 [35.6,43.3]
SigLIP-2 B/16	modify_color	618	39.2 [35.3,43.0]
SigLIP-2 L/16	modify_color	618	38.5 [34.7,42.3]
SigLIP-2 SO400M	modify_color	618	36.2 [32.5,40.0]
NegCLIP B/32	modify_color	618	30.6 [26.9,34.2]
CLoVe B/32	modify_color	618	27.7 [24.1,31.2]
DreamLIP B/16	modify_color	618	52.4 [48.5,56.4]
DFN H/14	modify_color	618	35.1 [31.3,38.9]
EVA-02 L/14	modify_color	618	26.5 [23.1,30.0]
MetaCLIP-2 H/14	modify_color	618	27.2 [23.7,30.7]
CLIP B/32	modify_pattern	303	6.3 [3.5,9.0]
CLIP B/16 LAION-2B	modify_pattern	303	14.5 [10.6,18.5]
CLIP H/14 LAION-2B	modify_pattern	303	9.6 [6.3,12.9]
SigLIP B/16	modify_pattern	303	20.1 [15.6,24.6]
SigLIP SO400M	modify_pattern	303	21.5 [16.8,26.1]
SigLIP-2 B/16	modify_pattern	303	17.8 [13.5,22.1]
SigLIP-2 L/16	modify_pattern	303	16.8 [12.6,21.0]
SigLIP-2 SO400M	modify_pattern	303	23.8 [19.0,28.6]
NegCLIP B/32	modify_pattern	303	8.6 [5.4,11.7]
CLoVe B/32	modify_pattern	303	8.9 [5.7,12.1]
DreamLIP B/16	modify_pattern	303	29.4 [24.2,34.5]
DFN H/14	modify_pattern	303	9.9 [6.5,13.3]
EVA-02 L/14	modify_pattern	303	14.2 [10.3,18.1]
MetaCLIP-2 H/14	modify_pattern	303	11.9 [8.2,15.5]
CLIP B/32	modify_material	483	1.7 [0.5,2.8]
CLIP B/16 LAION-2B	modify_material	483	6.8 [4.6,9.1]
CLIP H/14 LAION-2B	modify_material	483	8.3 [5.8,10.7]
SigLIP B/16	modify_material	483	11.2 [8.4,14.0]
SigLIP SO400M	modify_material	483	16.4 [13.1,19.7]
SigLIP-2 B/16	modify_material	483	10.8 [8.0,13.5]
SigLIP-2 L/16	modify_material	483	14.7 [11.5,17.9]
SigLIP-2 SO400M	modify_material	483	13.7 [10.6,16.7]
NegCLIP B/32	modify_material	483	4.3 [2.5,6.2]
CLoVe B/32	modify_material	483	3.3 [1.7,4.9]
DreamLIP B/16	modify_material	483	12.2 [9.3,15.1]
DFN H/14	modify_material	483	12.6 [9.7,15.6]
EVA-02 L/14	modify_material	483	8.9 [6.4,11.4]
MetaCLIP-2 H/14	modify_material	483	8.1 [5.6,10.5]
CLIP B/32	modify_shape	309	2.6 [0.8,4.4]
CLIP B/16 LAION-2B	modify_shape	309	8.7 [5.6,11.9]
CLIP H/14 LAION-2B	modify_shape	309	10.7 [7.2,14.1]
SigLIP B/16	modify_shape	309	10.0 [6.7,13.4]
SigLIP SO400M	modify_shape	309	12.3 [8.6,16.0]
SigLIP-2 B/16	modify_shape	309	11.0 [7.5,14.5]
SigLIP-2 L/16	modify_shape	309	14.6 [10.6,18.5]
SigLIP-2 SO400M	modify_shape	309	12.6 [8.9,16.3]
NegCLIP B/32	modify_shape	309	6.8 [4.0,9.6]
CLoVe B/32	modify_shape	309	4.5 [2.2,6.8]
DreamLIP B/16	modify_shape	309	14.6 [10.6,18.5]
DFN H/14	modify_shape	309	9.4 [6.1,12.6]
EVA-02 L/14	modify_shape	309	7.4 [4.5,10.4]
MetaCLIP-2 H/14	modify_shape	309	12.0 [8.4,15.6]
CLIP B/32	modify_transparency	300	0.7 [0.0,1.6]
CLIP B/16 LAION-2B	modify_transparency	300	1.7 [0.2,3.1]
CLIP H/14 LAION-2B	modify_transparency	300	0.7 [0.0,1.6]
SigLIP B/16	modify_transparency	300	1.3 [0.0,2.6]
SigLIP SO400M	modify_transparency	300	1.7 [0.2,3.1]
SigLIP-2 B/16	modify_transparency	300	2.3 [0.6,4.0]
SigLIP-2 L/16	modify_transparency	300	2.3 [0.6,4.0]
SigLIP-2 SO400M	modify_transparency	300	1.0 [0.0,2.1]
NegCLIP B/32	modify_transparency	300	1.0 [0.0,2.1]
CLoVe B/32	modify_transparency	300	0.7 [0.0,1.6]
DreamLIP B/16	modify_transparency	300	1.7 [0.2,3.1]
DFN H/14	modify_transparency	300	1.3 [0.0,2.6]
EVA-02 L/14	modify_transparency	300	0.3 [0.0,1.0]
MetaCLIP-2 H/14	modify_transparency	300	2.3 [0.6,4.0]
CLIP B/32	modify_object	303	5.9 [3.3,8.6]
CLIP B/16 LAION-2B	modify_object	303	9.9 [6.5,13.3]
CLIP H/14 LAION-2B	modify_object	303	14.2 [10.3,18.1]
SigLIP B/16	modify_object	303	8.9 [5.7,12.1]
SigLIP SO400M	modify_object	303	23.8 [19.0,28.6]
SigLIP-2 B/16	modify_object	303	18.8 [14.4,23.2]
SigLIP-2 L/16	modify_object	303	28.4 [23.3,33.5]
SigLIP-2 SO400M	modify_object	303	29.7 [24.6,34.8]
NegCLIP B/32	modify_object	303	7.3 [4.3,10.2]
CLoVe B/32	modify_object	303	5.0 [2.5,7.4]
DreamLIP B/16	modify_object	303	8.3 [5.2,11.3]
DFN H/14	modify_object	303	16.5 [12.3,20.7]
EVA-02 L/14	modify_object	303	19.5 [15.0,23.9]
MetaCLIP-2 H/14	modify_object	303	18.2 [13.8,22.5]
CLIP B/32	swap_color	312	2.2 [0.6,3.9]
CLIP B/16 LAION-2B	swap_color	312	1.9 [0.4,3.4]
CLIP H/14 LAION-2B	swap_color	312	1.6 [0.2,3.0]
SigLIP B/16	swap_color	312	3.2 [1.3,5.2]
SigLIP SO400M	swap_color	312	3.2 [1.3,5.2]
SigLIP-2 B/16	swap_color	312	5.1 [2.7,7.6]
SigLIP-2 L/16	swap_color	312	2.6 [0.8,4.3]
SigLIP-2 SO400M	swap_color	312	3.5 [1.5,5.6]
NegCLIP B/32	swap_color	312	3.2 [1.3,5.2]
CLoVe B/32	swap_color	312	2.3 [0.6,3.9]
DreamLIP B/16	swap_color	312	18.6 [14.3,22.9]
DFN H/14	swap_color	312	3.5 [1.5,5.6]
EVA-02 L/14	swap_color	312	1.6 [0.2,3.0]
MetaCLIP-2 H/14	swap_color	312	0.3 [0.0,0.9]
CLIP B/32	swap_material	303	0.0 [0.0,0.0]
CLIP B/16 LAION-2B	swap_material	303	1.7 [0.2,3.1]
CLIP H/14 LAION-2B	swap_material	303	2.0 [0.4,3.5]
SigLIP B/16	swap_material	303	3.0 [1.1,4.9]
SigLIP SO400M	swap_material	303	2.0 [0.4,3.5]
SigLIP-2 B/16	swap_material	303	4.3 [2.0,6.6]
SigLIP-2 L/16	swap_material	303	1.3 [0.0,2.6]
SigLIP-2 SO400M	swap_material	303	2.6 [0.8,4.4]
NegCLIP B/32	swap_material	303	0.3 [0.0,1.0]
CLoVe B/32	swap_material	303	0.0 [0.0,0.0]
DreamLIP B/16	swap_material	303	3.0 [1.1,4.9]
DFN H/14	swap_material	303	2.0 [0.4,3.5]
EVA-02 L/14	swap_material	303	1.0 [0.0,2.1]
MetaCLIP-2 H/14	swap_material	303	1.0 [0.0,2.1]
CLIP B/32	modify_count	300	1.7 [0.2,3.1]
CLIP B/16 LAION-2B	modify_count	300	0.7 [0.0,1.6]
CLIP H/14 LAION-2B	modify_count	300	2.3 [0.6,4.0]
SigLIP B/16	modify_count	300	2.0 [0.4,3.6]
SigLIP SO400M	modify_count	300	4.0 [1.8,6.2]
SigLIP-2 B/16	modify_count	300	3.7 [1.5,5.8]
SigLIP-2 L/16	modify_count	300	5.0 [2.5,7.5]
SigLIP-2 SO400M	modify_count	300	5.3 [2.8,7.9]
NegCLIP B/32	modify_count	300	1.7 [0.2,3.1]
CLoVe B/32	modify_count	300	0.7 [0.0,1.6]
DreamLIP B/16	modify_count	300	3.3 [1.3,5.4]
DFN H/14	modify_count	300	4.7 [2.3,7.1]
EVA-02 L/14	modify_count	300	4.0 [1.8,6.2]
MetaCLIP-2 H/14	modify_count	300	3.3 [1.3,5.4]
CLIP B/32	modify_presence	312	0.6 [0.0,1.5]
CLIP B/16 LAION-2B	modify_presence	312	1.0 [0.0,2.0]
CLIP H/14 LAION-2B	modify_presence	312	1.6 [0.2,3.0]
SigLIP B/16	modify_presence	312	1.0 [0.0,2.0]
SigLIP SO400M	modify_presence	312	1.0 [0.0,2.0]
SigLIP-2 B/16	modify_presence	312	1.9 [0.4,3.4]
SigLIP-2 L/16	modify_presence	312	2.9 [1.0,4.7]
SigLIP-2 SO400M	modify_presence	312	1.0 [0.0,2.0]
NegCLIP B/32	modify_presence	312	1.3 [0.0,2.5]
CLoVe B/32	modify_presence	312	0.0 [0.0,0.0]
DreamLIP B/16	modify_presence	312	0.6 [0.0,1.5]
DFN H/14	modify_presence	312	1.0 [0.0,2.0]
EVA-02 L/14	modify_presence	312	0.6 [0.0,1.5]
MetaCLIP-2 H/14	modify_presence	312	0.6 [0.0,1.5]
CLIP B/32	modify_spatial	297	0.3 [0.0,1.0]
CLIP B/16 LAION-2B	modify_spatial	297	0.7 [0.0,1.6]
CLIP H/14 LAION-2B	modify_spatial	297	0.0 [0.0,0.0]
SigLIP B/16	modify_spatial	297	0.3 [0.0,1.0]
SigLIP SO400M	modify_spatial	297	0.0 [0.0,0.0]
SigLIP-2 B/16	modify_spatial	297	0.0 [0.0,0.0]
SigLIP-2 L/16	modify_spatial	297	0.3 [0.0,1.0]
SigLIP-2 SO400M	modify_spatial	297	0.3 [0.0,1.0]
NegCLIP B/32	modify_spatial	297	0.3 [0.0,1.0]
CLoVe B/32	modify_spatial	297	0.7 [0.0,1.6]
DreamLIP B/16	modify_spatial	297	0.0 [0.0,0.0]
DFN H/14	modify_spatial	297	0.0 [0.0,0.0]
EVA-02 L/14	modify_spatial	297	0.7 [0.0,1.6]
MetaCLIP-2 H/14	modify_spatial	297	0.0 [0.0,0.0]
CLIP B/32	swap_spatial	300	0.0 [0.0,0.0]
CLIP B/16 LAION-2B	swap_spatial	300	0.0 [0.0,0.0]
CLIP H/14 LAION-2B	swap_spatial	300	0.0 [0.0,0.0]
SigLIP B/16	swap_spatial	300	0.0 [0.0,0.0]
SigLIP SO400M	swap_spatial	300	0.7 [0.0,1.6]
SigLIP-2 B/16	swap_spatial	300	0.7 [0.0,1.6]
SigLIP-2 L/16	swap_spatial	300	0.7 [0.0,1.6]
SigLIP-2 SO400M	swap_spatial	300	1.0 [0.0,2.1]
NegCLIP B/32	swap_spatial	300	0.3 [0.0,1.0]
CLoVe B/32	swap_spatial	300	0.0 [0.0,0.0]
DreamLIP B/16	swap_spatial	300	0.3 [0.0,1.0]
DFN H/14	swap_spatial	300	0.0 [0.0,0.0]
EVA-02 L/14	swap_spatial	300	1.0 [0.0,2.1]
MetaCLIP-2 H/14	swap_spatial	300	0.3 [0.0,1.0]
CLIP B/32	modify_action	303	7.9 [4.9,11.0]
CLIP B/16 LAION-2B	modify_action	303	17.2 [12.9,21.4]
CLIP H/14 LAION-2B	modify_action	303	26.1 [21.1,31.0]
SigLIP B/16	modify_action	303	22.1 [17.4,26.8]
SigLIP SO400M	modify_action	303	46.5 [40.9,52.2]
SigLIP-2 B/16	modify_action	303	21.1 [16.5,25.7]
SigLIP-2 L/16	modify_action	303	46.5 [40.9,52.2]
SigLIP-2 SO400M	modify_action	303	48.8 [43.2,54.5]
NegCLIP B/32	modify_action	303	13.2 [9.4,17.0]
CLoVe B/32	modify_action	303	6.9 [4.1,9.8]
DreamLIP B/16	modify_action	303	17.5 [13.2,21.8]
DFN H/14	modify_action	303	33.3 [28.0,38.6]
EVA-02 L/14	modify_action	303	25.1 [20.2,30.0]
MetaCLIP-2 H/14	modify_action	303	31.4 [26.1,36.6]
CLIP B/32	modify_pose	312	3.8 [1.7,6.0]
CLIP B/16 LAION-2B	modify_pose	312	5.4 [2.9,8.0]
CLIP H/14 LAION-2B	modify_pose	312	4.5 [2.2,6.8]
SigLIP B/16	modify_pose	312	10.6 [7.2,14.0]
SigLIP SO400M	modify_pose	312	9.6 [6.3,12.9]
SigLIP-2 B/16	modify_pose	312	10.3 [6.9,13.6]
SigLIP-2 L/16	modify_pose	312	10.3 [6.9,13.6]
SigLIP-2 SO400M	modify_pose	312	13.5 [9.7,17.2]
NegCLIP B/32	modify_pose	312	7.1 [4.2,9.9]
CLoVe B/32	modify_pose	312	5.1 [2.7,7.6]
DreamLIP B/16	modify_pose	312	14.1 [10.2,18.0]
DFN H/14	modify_pose	312	12.8 [9.1,16.5]
EVA-02 L/14	modify_pose	312	5.1 [2.7,7.6]
MetaCLIP-2 H/14	modify_pose	312	4.5 [2.2,6.8]
CLIP B/32	modify_state	438	0.5 [0.0,1.1]
CLIP B/16 LAION-2B	modify_state	438	0.7 [0.0,1.5]
CLIP H/14 LAION-2B	modify_state	438	1.1 [0.1,2.1]
SigLIP B/16	modify_state	438	0.9 [0.0,1.8]
SigLIP SO400M	modify_state	438	2.1 [0.7,3.4]
SigLIP-2 B/16	modify_state	438	2.3 [0.9,3.7]
SigLIP-2 L/16	modify_state	438	3.2 [1.5,4.8]
SigLIP-2 SO400M	modify_state	438	2.7 [1.2,4.3]
NegCLIP B/32	modify_state	438	1.1 [0.1,2.1]
CLoVe B/32	modify_state	438	0.2 [0.0,0.7]
DreamLIP B/16	modify_state	438	0.5 [0.0,1.1]
DFN H/14	modify_state	438	1.8 [0.6,3.1]
EVA-02 L/14	modify_state	438	0.7 [0.0,1.5]
MetaCLIP-2 H/14	modify_state	438	0.7 [0.0,1.5]
CLIP B/32	swap_role	135	0.0 [0.0,0.0]
CLIP B/16 LAION-2B	swap_role	135	0.0 [0.0,0.0]
CLIP H/14 LAION-2B	swap_role	135	0.0 [0.0,0.0]
SigLIP B/16	swap_role	135	1.5 [0.0,3.5]
SigLIP SO400M	swap_role	135	1.5 [0.0,3.5]
SigLIP-2 B/16	swap_role	135	0.7 [0.0,2.2]
SigLIP-2 L/16	swap_role	135	1.5 [0.0,3.5]
SigLIP-2 SO400M	swap_role	135	3.0 [0.1,5.8]
NegCLIP B/32	swap_role	135	2.2 [0.0,4.7]
CLoVe B/32	swap_role	135	0.7 [0.0,2.2]
DreamLIP B/16	swap_role	135	2.2 [0.0,4.7]
DFN H/14	swap_role	135	1.5 [0.0,3.5]
EVA-02 L/14	swap_role	135	3.0 [0.1,5.8]
MetaCLIP-2 H/14	swap_role	135	3.0 [0.1,5.8]

Fourteen dual encoders; short/medium/long captions pooled. Preserve the printed point estimate and interval. Role swaps have only 45 base pairs (135 pair-caption observations).

CompoSET paper v1 — task, tables and limitations

Edit and caption-length summary across 14 models

Table 10 averages Group scores across the fourteen models. These aggregate values are not individual model results.

Edit and caption-length summary across 14 models
Edit type	n per tier	Short Group %	Medium Group %	Long Group %
modify_color	206	35.3	34.7	30.5
modify_pattern	101	16.1	15.2	14.4
modify_material	161	10.2	10.0	8.3
modify_shape	103	11.5	10.6	7.3
modify_transparency	100	1.6	1.3	1.1
modify_object	101	15.3	15.7	14.9
swap_color	104	3.1	3.4	4.8
swap_material	101	1.2	1.8	2.1
modify_count	100	3.4	3.1	2.5
modify_presence	104	0.3	0.5	2.5
modify_spatial	99	0.4	0.1	0.3
swap_spatial	100	0.4	0.4	0.3
modify_action	101	27.2	27.3	23.4
modify_pose	104	10.4	9.3	5.3
modify_state	146	1.4	1.6	1.0
swap_role	45	1.4	0.8	2.2

Model-averaged subset diagnostics from Table 10; excluded from the primary score series.

CompoSET paper v1 — task, tables and limitations

200-instance generative slice

Table 11 uses a different scoring primitive from dual encoders. Open-weight models score caption likelihood; API models choose between options with order counterbalancing.

200-instance generative slice
Model	Scoring method	I2T %	T2I %	Group %
BLIP-2 (OPT-2.7B)	Open-weight (caption log-likelihood)	5.0	11.0	2.5
LLaVA-1.6 (Mistral-7B)	Open-weight (caption log-likelihood)	30.5	3.0	1.5
Qwen2-VL (7B Instruct)	Open-weight (caption log-likelihood)	36.5	0.5	0.5
Gemini 2.5 Flash	Frontier API (prompted pairwise, counterbalanced)	64.5	16.0	12.0
Claude Haiku 4.5	Frontier API (prompted pairwise, counterbalanced)	46.5	17.0	13.0
Claude Sonnet 4.6	Frontier API (prompted pairwise, counterbalanced)	71.5	47.0	42.5

Stratified 200-instance subset. API temperature 0; log-likelihood is length-normalized. I2T is the paper’s preferred diagnostic for the likelihood slice; Group is shown as reported, not mixed into overall results.

CompoSET paper v1 — task, tables and limitations

API models by caption length

Table 12 reports Group for each verbosity tier in the small API-model slice.

API models by caption length
Model	Short Group %	Medium Group %	Long Group %
Gemini 2.5 Flash	15.9	19.7	13.6
Claude Haiku 4.5	11.1	19.7	7.6
Claude Sonnet 4.6	44.4	46.5	36.4

Approximately 67 instances per tier; temperature 0 and order counterbalancing. These values condition on a subset and are not full-benchmark scores.

CompoSET paper v1 — task, tables and limitations

Original publication

Original paper v1 and author-linked release materials, reviewed 2 November 2026. Scores remain attributed to the benchmark publisher.

Reviewed 2026-11-02

Evidence available for this benchmark

This source record contains 22 numeric observations. The source distinguishes 22 model identities or configurations. A model may appear more than once; check the version, tools and run budget before comparing. Read results and row-level evidence →

Reported metric: Overall strict Group accuracy
Recorded unit: percent
Metric direction: higher is better

Start with these resources

Paper · task design and experiments · arxiv.org

Cited documents and dates (1)

CompoSET paper v1 · overall strict Group accuracy (Table 8) · 2026-10-30

Evaluation context and evidence

Source: Model reports
Publisher: Not recorded
Benchmark release: 2026-10-30
Collected: Not recorded
Modality: Not recorded

These are source-reported observations, not independent reruns by this site. Check the task version, evaluation protocol, unit and reporting date before comparing models. Results from different setups are not a universal ranking.

Model	Value	Unit	Protocol	Reported date and basis	Source
CLIP B/16 (LAION-2B)	7.8	percent	Paper v1 Table 8; full CompoSET release, all caption verbosity tiers; deterministic dual-encoder cosine similarity; strict bidirectional Group pass rate; paper-reported checkpoint identities in Table 4.	2026-10-30 · document_publication	CompoSET paper v1 · overall strict Group accuracy (Table 8)
CLIP B/16 (LAION-400M)	8.4	percent	Paper v1 Table 8; full CompoSET release, all caption verbosity tiers; deterministic dual-encoder cosine similarity; strict bidirectional Group pass rate; paper-reported checkpoint identities in Table 4.	2026-10-30 · document_publication	CompoSET paper v1 · overall strict Group accuracy (Table 8)
CLIP B/16 (OpenAI)	4.8	percent	Paper v1 Table 8; full CompoSET release, all caption verbosity tiers; deterministic dual-encoder cosine similarity; strict bidirectional Group pass rate; paper-reported checkpoint identities in Table 4.	2026-10-30 · document_publication	CompoSET paper v1 · overall strict Group accuracy (Table 8)
CLIP B/32	4.4	percent	Paper v1 Table 8; full CompoSET release, all caption verbosity tiers; deterministic dual-encoder cosine similarity; strict bidirectional Group pass rate; paper-reported checkpoint identities in Table 4.	2026-10-30 · document_publication	CompoSET paper v1 · overall strict Group accuracy (Table 8)
CLIP H/14 (LAION-2B)	8.4	percent	Paper v1 Table 8; full CompoSET release, all caption verbosity tiers; deterministic dual-encoder cosine similarity; strict bidirectional Group pass rate; paper-reported checkpoint identities in Table 4.	2026-10-30 · document_publication	CompoSET paper v1 · overall strict Group accuracy (Table 8)
CLIP L/14 (LAION-2B)	8.4	percent	Paper v1 Table 8; full CompoSET release, all caption verbosity tiers; deterministic dual-encoder cosine similarity; strict bidirectional Group pass rate; paper-reported checkpoint identities in Table 4.	2026-10-30 · document_publication	CompoSET paper v1 · overall strict Group accuracy (Table 8)
CLIP L/14 (OpenAI)	5.0	percent	Paper v1 Table 8; full CompoSET release, all caption verbosity tiers; deterministic dual-encoder cosine similarity; strict bidirectional Group pass rate; paper-reported checkpoint identities in Table 4.	2026-10-30 · document_publication	CompoSET paper v1 · overall strict Group accuracy (Table 8)
CLoVe B/32	5.5	percent	Paper v1 Table 8; full CompoSET release, all caption verbosity tiers; deterministic dual-encoder cosine similarity; strict bidirectional Group pass rate; paper-reported checkpoint identities in Table 4.	2026-10-30 · document_publication	CompoSET paper v1 · overall strict Group accuracy (Table 8)
DFN B/16	8.4	percent	Paper v1 Table 8; full CompoSET release, all caption verbosity tiers; deterministic dual-encoder cosine similarity; strict bidirectional Group pass rate; paper-reported checkpoint identities in Table 4.	2026-10-30 · document_publication	CompoSET paper v1 · overall strict Group accuracy (Table 8)
DFN H/14	10.8	percent	Paper v1 Table 8; full CompoSET release, all caption verbosity tiers; deterministic dual-encoder cosine similarity; strict bidirectional Group pass rate; paper-reported checkpoint identities in Table 4.	2026-10-30 · document_publication	CompoSET paper v1 · overall strict Group accuracy (Table 8)
DreamLIP B/16 (30M)	13.7	percent	Paper v1 Table 8; full CompoSET release, all caption verbosity tiers; deterministic dual-encoder cosine similarity; strict bidirectional Group pass rate; paper-reported checkpoint identities in Table 4.	2026-10-30 · document_publication	CompoSET paper v1 · overall strict Group accuracy (Table 8)
EVA-02 B/16	7.8	percent	Paper v1 Table 8; full CompoSET release, all caption verbosity tiers; deterministic dual-encoder cosine similarity; strict bidirectional Group pass rate; paper-reported checkpoint identities in Table 4.	2026-10-30 · document_publication	CompoSET paper v1 · overall strict Group accuracy (Table 8)
EVA-02 L/14	8.6	percent	Paper v1 Table 8; full CompoSET release, all caption verbosity tiers; deterministic dual-encoder cosine similarity; strict bidirectional Group pass rate; paper-reported checkpoint identities in Table 4.	2026-10-30 · document_publication	CompoSET paper v1 · overall strict Group accuracy (Table 8)
Jina-v4	8.5	percent	Paper v1 Table 8; full CompoSET release, all caption verbosity tiers; deterministic dual-encoder cosine similarity; strict bidirectional Group pass rate; paper-reported checkpoint identities in Table 4.	2026-10-30 · document_publication	CompoSET paper v1 · overall strict Group accuracy (Table 8)
MetaCLIP-2 H/14	8.9	percent	Paper v1 Table 8; full CompoSET release, all caption verbosity tiers; deterministic dual-encoder cosine similarity; strict bidirectional Group pass rate; paper-reported checkpoint identities in Table 4.	2026-10-30 · document_publication	CompoSET paper v1 · overall strict Group accuracy (Table 8)
NegCLIP B/32	7.0	percent	Paper v1 Table 8; full CompoSET release, all caption verbosity tiers; deterministic dual-encoder cosine similarity; strict bidirectional Group pass rate; paper-reported checkpoint identities in Table 4.	2026-10-30 · document_publication	CompoSET paper v1 · overall strict Group accuracy (Table 8)
SigLIP B/16	10.2	percent	Paper v1 Table 8; full CompoSET release, all caption verbosity tiers; deterministic dual-encoder cosine similarity; strict bidirectional Group pass rate; paper-reported checkpoint identities in Table 4.	2026-10-30 · document_publication	CompoSET paper v1 · overall strict Group accuracy (Table 8)
SigLIP L/16-384	12.7	percent	Paper v1 Table 8; full CompoSET release, all caption verbosity tiers; deterministic dual-encoder cosine similarity; strict bidirectional Group pass rate; paper-reported checkpoint identities in Table 4.	2026-10-30 · document_publication	CompoSET paper v1 · overall strict Group accuracy (Table 8)
SigLIP SO400M/14	13.5	percent	Paper v1 Table 8; full CompoSET release, all caption verbosity tiers; deterministic dual-encoder cosine similarity; strict bidirectional Group pass rate; paper-reported checkpoint identities in Table 4.	2026-10-30 · document_publication	CompoSET paper v1 · overall strict Group accuracy (Table 8)
SigLIP-2 B/16	11.3	percent	Paper v1 Table 8; full CompoSET release, all caption verbosity tiers; deterministic dual-encoder cosine similarity; strict bidirectional Group pass rate; paper-reported checkpoint identities in Table 4.	2026-10-30 · document_publication	CompoSET paper v1 · overall strict Group accuracy (Table 8)
SigLIP-2 L/16-384	13.6	percent	Paper v1 Table 8; full CompoSET release, all caption verbosity tiers; deterministic dual-encoder cosine similarity; strict bidirectional Group pass rate; paper-reported checkpoint identities in Table 4.	2026-10-30 · document_publication	CompoSET paper v1 · overall strict Group accuracy (Table 8)
SigLIP-2 SO400M/14	13.9	percent	Paper v1 Table 8; full CompoSET release, all caption verbosity tiers; deterministic dual-encoder cosine similarity; strict bidirectional Group pass rate; paper-reported checkpoint identities in Table 4.	2026-10-30 · document_publication	CompoSET paper v1 · overall strict Group accuracy (Table 8)

What the test isolates

Dataset and construction

Read Group accuracy correctly

Separate the reported experiments

What the results cannot establish

Find the original release

Full benchmark: 22 dual encoders

Edit subsets: Group by model and edit type

Edit and caption-length summary across 14 models

200-instance generative slice

API models by caption length

Original publication

Evidence available for this benchmark

Start with these resources

Evaluation context and evidence

Reported results 22

Explore related benchmarks

Put this benchmark in context

Six new benchmark pages: context, agents and executable tasks

Six Navers Lab / Einsia benchmarks and the work they test

AI benchmarks and leaderboards are different things

How to compare AI benchmark scores without mixing protocols

Four new agent benchmarks: adaptation, safety and video

Continue exploring