Original paper v1, submitted 4 November 2026; the 1,000-question offline suite, Tables 2 and 12. Reviewed 5 November; execution dates and exact hosted snapshots are unspecified.
Visual judgments before physical execution
The suite contains 600 image questions and 400 video questions. Its final questions draw on 410 images and 400 videos; 1,000 questions therefore does not mean 1,000 independent scenes. Experts authored and reviewed the questions and labels.
| Task | Questions |
|---|---|
| AFF · affordance grounding | 150 |
| TRAJ · trajectory planning | 150 |
| ERI · equipment and reagent identification | 60 |
| ESS · safety and standardization | 60 |
| SR · state recognition | 60 |
| EAS · apparatus setup | 60 |
| EOP · experimental operations | 60 |
| TSR · temporal sequence reasoning | 150 |
| SVOR · short-video operations | 125 |
| MVTS · mid-video steps | 75 |
| LVPR · long-video procedure reasoning | 50 |
Task counts total 1,000; the same source image can support different questions.
One turn with standardized visual input
Models use task-specific prompts, temperature 0.7 when adjustable, original-resolution images and 8/32/128 sampled frames for short/mid/long video. Long videos use 32 four-frame grids. No live robot is controlled. Model labels preserve the paper spelling; exact hosted revisions and experiment dates remain unknown.
A weighted score, not a success percentage
Overall combines Actionability, Knowledge, Experimentation and Reasoning with weights 0.25/0.15/0.35/0.25. Experimentation weights image and video task groups 2:5. Component metrics include multiple-choice accuracy, spatial IoU, a transformed trajectory distance and expert semantic marking of fill-in answers.
All task scores, with human reference
Table 2 retains every evaluated model and the separate human row. Human results average three chemistry experts who did not construct the dataset and answered all questions. The AFF/TRAJ columns measure spatial tasks; neither is a physical execution rate.
| Model/reference | AFF | TRAJ | ERI | ESS | SR | EAS | EOP | SVOR | MVTS | TSR | LVPR | Overall |
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Human | 88.0 | 91.0 | 96.7 | 95.0 | 93.3 | 93.3 | 96.7 | 94.4 | 92.0 | 95.0 | 96.0 | 93.3 |
| ChemVLM-8B | 2.1 | 21.0 | 33.3 | 40.0 | 21.7 | 36.7 | 33.3 | 36.0 | 40.0 | 37.9 | 24.0 | 28.4 |
| ChemVLM-26B | 3.1 | 27.0 | 23.3 | 25.0 | 31.7 | 36.7 | 33.3 | 37.6 | 42.7 | 42.4 | 28.0 | 30.1 |
| Intern-S1 | 3.4 | 24.6 | 40.0 | 25.0 | 48.3 | 33.3 | 48.3 | 61.6 | 60.0 | 41.4 | 54.8 | 40.5 |
| Intern-S1-Mini | 1.7 | 22.2 | 43.3 | 38.3 | 43.3 | 30.0 | 41.7 | 46.4 | 56.0 | 24.0 | 52.7 | 35.2 |
| Intern-S1-Pro | 2.9 | 18.0 | 45.0 | 38.3 | 51.7 | 30.0 | 71.7 | 72.0 | 65.3 | 37.0 | 62.2 | 44.0 |
| ACE-Brain-0-8B | 2.4 | 23.0 | 33.3 | 76.7 | 31.7 | 63.3 | 55.0 | 68.8 | 65.3 | 54.3 | 40.0 | 44.7 |
| Pelican-VL-1.0-7B | 10.9 | 24.2 | 26.7 | 41.7 | 30.0 | 55.0 | 63.3 | 50.4 | 48.0 | 28.5 | 32.8 | 35.2 |
| RoboBrain2.5-8B | 2.3 | 30.9 | 41.7 | 35.0 | 48.3 | 43.3 | 58.3 | 74.4 | 61.3 | 55.3 | 47.5 | 45.3 |
| RynnBrain-8B | 3.2 | 40.7 | 28.3 | 45.0 | 28.3 | 45.0 | 43.3 | 52.8 | 46.7 | 47.3 | 47.3 | 39.2 |
| RynnBrain-CoP-8B | 28.6 | 38.6 | 25.0 | 41.7 | 30.0 | 28.3 | 41.7 | 56.8 | 50.7 | 53.8 | 47.7 | 42.9 |
| RynnBrain-Plan-8B | 7.0 | 28.0 | 31.7 | 48.3 | 26.7 | 40.0 | 60.0 | 52.0 | 37.3 | 47.2 | 45.3 | 37.4 |
| MiMo-Embodied-7B | 9.8 | 28.9 | 23.3 | 38.3 | 43.3 | 33.3 | 51.7 | 57.6 | 48.0 | 45.9 | 48.0 | 39.3 |
| GPT-5.5 | 32.7 | 47.2 | 55.0 | 28.3 | 61.7 | 63.3 | 71.7 | 72.0 | 49.3 | 65.0 | 64.7 | 55.4 |
| GPT-5.4 | 26.8 | 46.1 | 48.3 | 58.3 | 48.3 | 53.3 | 71.7 | 80.8 | 57.3 | 70.3 | 64.2 | 57.2 |
| Claude Opus 4.7 | 34.5 | 48.9 | 45.0 | 55.0 | 61.7 | 65.0 | 63.3 | 80.8 | 64.0 | 67.2 | 62.2 | 59.2 |
| Claude Sonnet 4.6 | 18.3 | 46.3 | 41.7 | 38.3 | 45.0 | 45.0 | 56.7 | 76.8 | 64.0 | 65.8 | 54.8 | 52.1 |
| Gemini 2.5 Pro | 7.4 | 44.4 | 43.3 | 23.3 | 53.3 | 20.0 | 45.0 | 47.2 | 40.0 | 46.3 | 67.0 | 40.8 |
| Gemini 3.1 Pro Preview | 4.7 | 29.6 | 66.7 | 33.3 | 60.0 | 65.0 | 70.0 | 82.4 | 64.0 | 85.9 | 59.2 | 55.5 |
| Grok-4.20-Non-Reasoning | 8.7 | 39.0 | 31.7 | 38.3 | 31.7 | 25.0 | 56.7 | 65.6 | 53.3 | 63.6 | 54.0 | 44.7 |
| Grok-4.20-Reasoning | 9.7 | 43.3 | 38.3 | 33.3 | 61.7 | 31.7 | 56.7 | 71.2 | 60.0 | 83.1 | 67.7 | 52.9 |
| MiMo-V2-Omni | 9.2 | 42.6 | 38.3 | 41.7 | 40.0 | 36.7 | 58.3 | 76.0 | 66.7 | 51.9 | 58.2 | 48.8 |
| Qwen3.6-Plus | 4.9 | 41.1 | 41.7 | 36.7 | 41.7 | 43.3 | 71.7 | 76.8 | 62.7 | 68.4 | 58.3 | 50.8 |
| Kimi-K2.5 | 6.7 | 42.5 | 43.3 | 63.3 | 43.3 | 46.7 | 71.7 | 79.2 | 62.7 | 60.3 | 59.3 | 52.2 |
| Kimi-K2.6 | 5.9 | 41.8 | 46.7 | 41.7 | 45.0 | 46.7 | 71.7 | 79.2 | 64.0 | 64.4 | 60.3 | 52.0 |
| MiMo-V2.5 | 11.3 | 40.1 | 45.0 | 41.7 | 45.0 | 36.7 | 61.7 | 72.0 | 61.3 | 51.0 | 57.8 | 48.2 |
| Qwen3-VL-8B | 4.1 | 26.8 | 45.0 | 41.7 | 56.7 | 45.0 | 60.0 | 75.2 | 66.7 | 54.2 | 50.2 | 47.1 |
| InternVL3.5-241B-A28B | 5.2 | 28.1 | 45.0 | 28.3 | 53.3 | 38.3 | 43.3 | 59.2 | 58.7 | 40.5 | 59.5 | 41.8 |
| Qwen3.5-397B-A17B | 3.8 | 37.3 | 50.0 | 53.3 | 45.0 | 65.0 | 71.7 | 79.2 | 65.3 | 67.9 | 64.7 | 54.0 |
Original 0–100 task scores; Human is a separate three-expert reference, not a model.
Capability profiles
A strong aggregate can conceal a weak capability. These are the original capability values, calculated before rounding; no values were recomputed from displayed task scores.
| Model | Actionability | Knowledge | Experimentation | Reasoning | Overall |
|---|---|---|---|---|---|
| ChemVLM-8B | 11.6 | 31.7 | 37.1 | 31.0 | 28.4 |
| ChemVLM-26B | 15.1 | 26.7 | 38.7 | 35.2 | 30.1 |
| Intern-S1 | 14.0 | 37.8 | 55.1 | 48.1 | 40.5 |
| Intern-S1-Mini | 12.0 | 41.7 | 46.8 | 38.3 | 35.2 |
| Intern-S1-Pro | 10.5 | 45.0 | 63.6 | 49.6 | 44.0 |
| ACE-Brain-0-8B | 12.7 | 47.2 | 64.8 | 47.2 | 44.7 |
| Pelican-VL-1.0-7B | 17.6 | 32.8 | 52.0 | 30.7 | 35.2 |
| RoboBrain2.5-8B | 16.6 | 41.7 | 63.0 | 51.4 | 45.3 |
| RynnBrain-8B | 21.9 | 33.9 | 48.1 | 47.3 | 39.2 |
| RynnBrain-CoP-8B | 33.6 | 32.2 | 48.4 | 50.7 | 42.9 |
| RynnBrain-Plan-8B | 17.5 | 35.6 | 46.2 | 46.3 | 37.4 |
| MiMo-Embodied-7B | 19.4 | 35.0 | 49.9 | 47.0 | 39.3 |
| GPT-5.5 | 40.0 | 48.3 | 62.6 | 64.8 | 55.4 |
| GPT-5.4 | 36.4 | 51.7 | 67.2 | 67.2 | 57.2 |
| Claude Opus 4.7 | 41.7 | 53.9 | 70.0 | 64.7 | 59.2 |
| Claude Sonnet 4.6 | 32.3 | 41.7 | 64.8 | 60.3 | 52.1 |
| Gemini 2.5 Pro | 25.9 | 40.0 | 40.4 | 56.7 | 40.8 |
| Gemini 3.1 Pro Preview | 17.2 | 53.3 | 71.6 | 72.6 | 55.5 |
| Grok-4.20-Non-Reasoning | 23.9 | 33.9 | 54.1 | 58.8 | 44.7 |
| Grok-4.20-Reasoning | 26.5 | 44.4 | 59.5 | 75.4 | 52.9 |
| MiMo-V2-Omni | 25.9 | 40.0 | 64.5 | 55.1 | 48.8 |
| Qwen3.6-Plus | 23.0 | 40.0 | 66.2 | 63.4 | 50.8 |
| Kimi-K2.5 | 24.6 | 50.0 | 67.6 | 59.8 | 52.2 |
| Kimi-K2.6 | 23.8 | 44.4 | 68.0 | 62.4 | 52.0 |
| MiMo-V2.5 | 25.7 | 43.9 | 61.7 | 54.4 | 48.2 |
| Qwen3-VL-8B | 15.4 | 47.8 | 65.7 | 52.2 | 47.1 |
| InternVL3.5-241B-A28B | 16.6 | 42.2 | 53.8 | 50.0 | 41.8 |
| Qwen3.5-397B-A17B | 20.6 | 49.4 | 71.1 | 66.3 | 54.0 |
| Mean (28 models) | 22.2 | 41.6 | 58.0 | 53.8 | 45.5 |
Includes the original Mean row; the mean is not an extra model observation.
Different question formats
Coordinate, multiple-choice and fill-in tasks have different scoring and content. Ratios in this table compare task groups, not isolated format robustness.
| Model | Coordinate | MCQ | Fill-in | Coord./MCQ | Fill-in/MCQ | Coord./Fill-in |
|---|---|---|---|---|---|---|
| ChemVLM-8B | 11.6 | 34.9 | 24.0 | 0.3 | 0.7 | 0.5 |
| GPT-5.5 | 40.0 | 58.3 | 64.7 | 0.7 | 1.1 | 0.6 |
| Claude Opus 4.7 | 41.7 | 62.8 | 62.2 | 0.7 | 1.0 | 0.7 |
| Qwen3.5-397B-A17B | 20.6 | 62.2 | 64.7 | 0.3 | 1.0 | 0.3 |
| RoboBrain2.5-8B | 16.6 | 52.2 | 47.5 | 0.3 | 0.9 | 0.3 |
MCQ is the unweighted macro-average of eight MCQ task types. Ratios are descriptive.
Frame-count sensitivity
Only Qwen3.5-397B-A17B is varied in this ablation. More frames do not improve every task, so frame settings remain part of the comparison contract.
| Setting | Short/mid/long frames | S-TSR | SVOR | M-TSR | MVTS | L-TSR | LVPR |
|---|---|---|---|---|---|---|---|
| Fewer Frames | 4 / 16 / 64 | 82.9 | 80.8 | 54.0 | 61.3 | 66.7 | 59.7 |
| Standard Frames main | 8 / 32 / 128 | 85.7 | 79.2 | 48.0 | 65.3 | 70.0 | 64.7 |
| More Frames | 16 / 48 / 192 | 88.6 | 80.8 | 52.0 | 60.0 | 73.3 | 55.3 |
One model ablation; long-video frames form four-frame grids.
Score and rank uncertainty
The source reports 10,000 paired, task-stratified bootstrap resamples. These intervals describe sampling uncertainty, not new independent model runs. Overlapping ranges caution against treating small differences as decisive.
| Model | Overall | 95% CI | 95% rank range |
|---|---|---|---|
| Claude Opus 4.7 | 59.2 | [56.7, 61.8] | 1–2 |
| GPT-5.4 | 57.2 | [54.5, 59.8] | 1–4 |
| Gemini 3.1 Pro Preview | 55.5 | [53.0, 57.9] | 2–6 |
| GPT-5.5 | 55.4 | [52.8, 58.0] | 2–6 |
| Qwen3.5-397B-A17B | 54.0 | [51.4, 56.6] | 3–8 |
| Grok-4.20-Reasoning | 52.9 | [50.4, 55.5] | 4–10 |
| Kimi-K2.5 | 52.2 | [49.6, 54.9] | 5–10 |
| Claude Sonnet 4.6 | 52.1 | [49.4, 54.7] | 5–10 |
| Kimi-K2.6 | 52.0 | [49.5, 54.6] | 5–10 |
| Qwen3.6-Plus | 50.8 | [48.2, 53.4] | 7–11 |
| MiMo-V2-Omni | 48.8 | [46.2, 51.5] | 10–13 |
| MiMo-V2.5 | 48.2 | [45.5, 50.9] | 10–14 |
| Qwen3-VL-8B | 47.1 | [44.5, 49.6] | 11–15 |
| RoboBrain2.5-8B | 45.3 | [42.7, 47.9] | 13–18 |
| ACE-Brain-0-8B | 44.7 | [42.2, 47.4] | 13–18 |
| Grok-4.20-Non-Reasoning | 44.7 | [42.1, 47.4] | 13–18 |
| Intern-S1-Pro | 44.0 | [41.5, 46.6] | 14–19 |
| RynnBrain-CoP-8B | 42.9 | [40.1, 45.6] | 15–21 |
| InternVL3.5-241B-A28B | 41.8 | [39.1, 44.6] | 16–22 |
| Gemini 2.5 Pro | 40.8 | [38.1, 43.5] | 18–23 |
| Intern-S1 | 40.5 | [37.9, 43.1] | 18–23 |
| MiMo-Embodied-7B | 39.3 | [36.5, 42.0] | 19–24 |
| RynnBrain-8B | 39.2 | [36.6, 41.9] | 19–24 |
| RynnBrain-Plan-8B | 37.4 | [34.7, 40.2] | 22–26 |
| Intern-S1-Mini | 35.2 | [32.5, 37.9] | 24–26 |
| Pelican-VL-1.0-7B | 35.2 | [32.6, 37.7] | 24–26 |
| ChemVLM-26B | 30.1 | [27.5, 32.5] | 27–28 |
| ChemVLM-8B | 28.4 | [25.8, 31.0] | 27–28 |
95% intervals from paired task-stratified resampling; smaller rank numbers are better.
Aggregation sensitivity
Alternative weightings largely preserve the top sets in this sample. This does not turn the composite into a physical deployment metric.
| Aggregation | Scheme | Spearman | Kendall | Top-5 | Top-10 |
|---|---|---|---|---|---|
| Equal-column macro | 13 score columns, equal weights | 0.994 | 0.952 | 5/5 | 10/10 |
| Equal-group macro | 3 task groups, equal weights | 0.993 | 0.947 | 5/5 | 10/10 |
| Sample-weighted micro | Weighted by question count | 0.996 | 0.963 | 5/5 | 10/10 |
| Rank-based | Mean within-column model rank | 0.980 | 0.889 | 5/5 | 10/10 |
Top-5 and Top-10 show overlap with the official aggregation; ties use average ranks.
What the benchmark cannot establish
Single-turn offline answers do not demonstrate closed-loop control, recovery or safe laboratory deployment. Public source media may overlap with training data even though questions and labels are new. Original source licenses and access terms still apply; this page links the paper and does not redistribute laboratory media.