Terminal-Bench 4.0, pinned to the public v4.0.0 release. Earlier versions retain separate records.
What it measures
Terminal-Bench 4 evaluates agents completing computer tasks through a terminal. The v4.0.0 manifest contains 66 tasks. Each combines a starting environment, an instruction and executable verification. This measures a model working inside an agent, including its ability to investigate, use tools and finish the job.
Version 4.0.0 dataset manifest · Harbor task structure, trials and rewards · Version 4.0.0 project overview and running instructions
What the agent does
Tasks span software, science, machine learning, operations, security, hardware and media. Examples include repairing database recovery and making batched model evaluation preserve single-example behavior. The task's main skill determines its category; using Python alone does not make a task software engineering. Subcategories can grow.
Version 4.0.0 taxonomy · Version 4.0.0 task: WAL recovery ordering · Version 4.0.0 task: batched evaluation parity
How results are checked
The official leaderboard reports resolution rate. Task verifiers check delivered behavior or artifacts and produce a reward. The reviewed task definitions use a separate verifier environment. A final answer saying the job is done is insufficient: the implementation must satisfy the task's checks. Read scores with the model, agent and run configuration.
Terminal-Bench official homepage — 4.0 and resolution rate · Harbor task structure, trials and rewards · WAL recovery ordering: resources and verifier configuration · Batched evaluation parity: resources and verifier configuration · Version 4.0.0 project overview and running instructions
When it is useful
Use it to examine whether an agent can carry work through an unfamiliar environment, rather than only propose a patch. Inspect relevant task definitions before choosing an evaluation: software repair, ML infrastructure and specialist scientific work exercise different skills.
Version 4.0.0 project overview and running instructions · Version 4.0.0 taxonomy · Version 4.0.0 task: WAL recovery ordering · Version 4.0.0 task: batched evaluation parity
What to keep separate
Pin the version: Terminal-Bench evolves, and 2.x, 3.0, Hard and 4.0 results are not interchangeable. Version 4 includes GPU and multi-container work, so infrastructure matters. Task sources and verification code are public; isolation during a run is different from a private dataset.
Version 4.0.0 project overview and running instructions · Version 4.0.0 task collection — GPU and multi-container support · Terminal-Bench v4.0.0 task sources · WAL recovery ordering: resources and verifier configuration