Four new agent benchmarks: adaptation, safety and video

AI Benchmark Radar / Equality Machine · Published 2026-10-27 · Reviewed 2026-10-27

This discovery batch includes four newly published agent benchmarks. They ask different practical questions: can an agent recover after tools change, reuse skills after a website changes, finish work safely, and deliver a usable video? A discovery notification counts newly seen artifacts; it does not establish that every item is a distinct new benchmark. This is a reading of the original publications, not an independent evaluation run.

Start with the failure you need to measure

A familiar task can fail because its tools changed, its interface moved, its execution crossed a safety boundary, or its final artifact never arrived. These benchmarks expose different parts of that workflow. Use the comparison below to select evidence for your application; it is not a combined ranking.

Benchmark	Question	Read first
Δτ-Bench	Can workflows adapt as tools evolve?	Task success and path efficiency
SkillShift-Bench	Do learned skills survive website changes?	Success and failures by shift condition
V³AST-Bench	Can capable agents avoid verified harm?	Capability controls alongside safety
VibeVideoBench	Does a usable video arrive on time?	Delivery, requirements and quality separately

Δτ-Bench — original paper, v1 · SkillShift-Bench — original paper, v1 · V³AST-Bench — original paper, v1; evaluation and limitations · VibeVideoBench — original paper, v1; protocol and limitations

Δτ-Bench: a working plan can become outdated

The benchmark covers 1,588 tasks across 42 domains and six successive tool states. User goals remain achievable while interfaces and available routes change. Its released split has 1,191 training tasks and 397 test tasks per version. SR measures whether the required environment outcome was reached; success-weighted path efficiency also asks how economically it was reached.

Read the data revision before comparing scores: the official repository says the released dataset includes corrections beyond the paper's experimental version. The published code and separately distributed executable data support reproduction, but a fresh run need not reproduce the paper's exact numbers.

Δτ-Bench — original paper, v1 · ZJUNLP — Δτ-Bench code, data scope and metrics

SkillShift-Bench: learning a skill is only the beginning

SkillShift follows 355 task families across self-hosted GitLab, Magento and WordPress systems, each under six conditions. It separates changes in invocation context, perceptual grounding and execution feasibility. Compare task success together with interaction cost, skill use and skill failure, rather than treating all website changes as one condition.

Its site publishes tasks, environments, reproduction instructions and aggregate results, but withholds individual task judgments and execution traces. Published baselines also use different backbones; keep those settings separate when reading the leaderboard.

SkillShift-Bench — original paper, v1 · SkillShift-Bench — official project and leaderboard · SkillShift-Bench — official code and release scope

V³AST-Bench: no harm does not automatically mean safe competence

V³AST pairs 212 risk tasks with 212 benign capability controls. Executable checks inspect consequences and task outcomes, across nine risk types and two attack sources. Capability-conditioned safety helps distinguish a capable agent that avoids harm from an agent that never managed the underlying workflow.

The paper evaluates model–harness combinations, not model names alone. Its Codex and Claude Code experiments use permissive autonomous settings inside dedicated task containers. These measurements do not describe the harnesses' restrictive safety modes. A failed benign control can also reflect over-refusal, so read capability coverage alongside conditional safety.

V³AST-Bench — original paper, v1; evaluation and limitations · Alibaba-AAIG — VAST-Bench official tasks and code

VibeVideoBench: the video must actually arrive

The suite contains 239 tasks in 12 categories: 209 English and 30 Chinese. Agents have one hour to produce code-rendered video and editable sources; each task specifies its audio requirements. Image- and video-generation services are excluded. Program checks assess delivery, task-specific checks assess requirements, and separate quality dimensions assess the result.

Delivered-only quality can hide missing videos. Read it alongside delivery and strict task success, or the separately reported no-video-equals-zero view. Each system has one scored run per task; judges only moderately agree on individual videos. The suite mainly measures adherence to a brief, not creativity in isolation.

VibeVideoBench — original paper, v1; protocol and limitations

Three notification entries need a different reading

The DeepSeek V4.1 Flash-1017 item is a safety evaluation report covering 34 evaluations, including five in-house suites. The report is not itself evidence of 34 newly introduced benchmarks.

SegDeep is a versioned dataset record. Its Zenodo metadata gives a publication date of August 24, 2026, while the record was created on October 26 at 20:50 UTC (October 27 at 04:50 in Beijing). The concept DOI ending 22086397 and version DOI ending 22086398 refer to the same dataset family; neither a newly discovered record nor its second identifier establishes a new dataset release.

VPR-Space defines a new evaluation from existing SLAM datasets. Its paper labels the release preliminary, and the authors' repository was unavailable when checked. We retain the paper link without claiming that the complete runnable benchmark is already downloadable. These examples show why a discovery notification count differs from a count of verified new benchmarks.

DeepSeek V4.1 Flash-1017 — safety evaluation report · SegDeep — version 1.0 record, dates and concept DOI · VPR-Space — original paper and preliminary release scope