Author repository at commit 79d00b3 (2 November 2026 UTC): 11 selected episodes, with the separately dated 29 October five-episode pilot. Source revision dates are not experiment dates.
Different annotations can both be correct
The evaluator accepts a range around each action boundary and allows optional subdivisions. A prediction is penalized for missing a required boundary or inventing an unsupported split, not merely for differing from one exact timestamp. Description evaluation uses fixed reference intervals, so segmentation and caption quality are measured separately.
11 selected videos, 221 fixed intervals
The current reference includes 11 selected HomER episodes, 221 fixed description intervals and 325 required details. The five-episode pilot covered 104 intervals and 145 required details. These selected clips do not stand for all of WGO-Bench or all robot-training videos.
Pinned author README · current results and methods · Original five-episode pilot README · 29 October 2026
Inputs differ for the native video model
API candidates receive timestamped 8-fps frames, resized to 512 pixels and grouped into 4-by-2 contact sheets. Segmentation uses a 10-second scoring core plus one second of surrounding context. NVIDIA FLORA instead uses its native TempSeg and Captioner pipeline: 8-fps source video, internal 4-fps temporal sampling and a guide condensed to at most 2,000 characters. Kimi K3 and K3.1 Pro have a 1,024-token reasoning cap. These settings remain attached to the results.
Pinned author README · current results and methods · Eleven-episode input, guide and reference hashes · Kimi K3.1 Pro · input and reasoning-cap configuration
The combined score multiplies two F1 values
Segmentation F1 aggregates boundary hits and errors across selected videos. A blinded text-only GPT-6 Astra judge compares generated descriptions with required and optional reference details, without seeing the video. Unsupported claims and missing required details affect description F1. Overall F1 is the product of these two micro F1 scores; it is a 0–1 score, not a percentage of whole videos passed.
Ten reported configurations on eleven videos
The table preserves the authors’ rounded README values at the November 2 source revision. Full-precision counts and per-episode diagnostics are linked in the original result archive. The older Qwen3-VL Instruct run is archived but absent from this current README comparison; no value is guessed for it here.
| Model | Segmentation F1 | Description F1 | Overall F1 |
|---|---|---|---|
| GPT-6 Astra | 0.773 | 0.709 | 0.548 |
| GPT-6.1 Sol | 0.756 | 0.676 | 0.511 |
| GPT-6.1 Terra | 0.740 | 0.668 | 0.494 |
| Gemini 4 Argon | 0.694 | 0.672 | 0.466 |
| NVIDIA FLORA | 0.529 | 0.670 | 0.354 |
| Claude Opus 5.5 | 0.548 | 0.590 | 0.323 |
| Claude Sonnet 5.5 | 0.520 | 0.489 | 0.254 |
| Kimi K3.1 Pro (1,024 reasoning tokens) | 0.523 | 0.414 | 0.217 |
| GPT-6 Luna | 0.474 | 0.432 | 0.204 |
| Kimi K3 (1,024 reasoning tokens) | 0.434 | 0.374 | 0.162 |
The overall value is reproduced as printed, not recalculated from already-rounded components. FLORA uses different video handling; Kimi has a reasoning cap.
Pinned author README · current results and methods · Original model outputs, judge decisions and results archive
Keep the earlier pilot separate
The five-episode pilot used an earlier guide and a stricter description judge. It has not been rerun under the updated rubric. Its results are retained under a separate instrument, not treated as model progress on the eleven-episode task. Gemini refused two windows, counted as missing annotations. Opus used OpenRouter in the pilot and the Anthropic API in the newer run.
| Model | Segmentation F1 | Description F1 | Overall F1 | Refused windows |
|---|---|---|---|---|
| GPT-6 Astra | 0.702 | 0.629 | 0.442 | 0 |
| GPT-6.1 Sol | 0.708 | 0.598 | 0.423 | 0 |
| Gemini 4 Argon | 0.584 | 0.609 | 0.355 | 2 |
| Claude Opus 5.5 | 0.542 | 0.495 | 0.268 | 0 |
| GPT-6 Luna | 0.491 | 0.362 | 0.178 | 0 |
Different task coverage and judging rules from the eleven-episode comparison.
Original five-episode pilot README · 29 October 2026 · Pinned author README · current results and methods
A model judge and a small selected sample
GPT-6 Astra appears both as a candidate and as the description judge. Blinding model identity does not remove model-judge bias or incomplete reference details. Contact sheets also differ from native-video input. This release provides reproducible materials, not independent evidence that the ranking holds across all activities or cameras.
Inspect the original guide, inputs and outputs
The author repository contains the runner, reference metadata, guide snapshots, model outputs, judge decisions and score archives. Videos are linked externally rather than bundled; the README identifies the upstream WGO-Bench video license as CC BY-NC-SA 4.0. The date attached to each score here is the cited author source revision. Exact execution timestamps and an initial benchmark publication date are not inferred. Radar reviewed the artifacts on November 4 and did not rerun the models.
Pinned author README · current results and methods · Eleven-episode input, guide and reference hashes · Original model outputs, judge decisions and results archive