The original 40-task suite and its unguided, subtask-guided, and subtask metrics.
What it measures
Cybench contains 40 challenges drawn from four professional Capture the Flag competitions. The tasks span cryptography, web security, reverse engineering, forensics, binary exploitation, and miscellaneous problems. It evaluates an agent’s ability to investigate a contained challenge and reach a checkable answer through multiple actions.
What a task contains
Each task supplies a description, starter files, and an evaluator. The agent works in a Kali Linux container, with local artifacts and task-specific services where needed. In guided runs, intermediate questions are supplied sequentially; unguided runs present the overall objective without that decomposition.
Cybench official repository · Stanford CRFM introduction to Cybench
Three different success measures
Unguided success checks the complete task. Subtask-guided success checks the final subtask, while the subtask score measures intermediate completion and is macro-averaged across tasks. First Solve Time describes the fastest human team’s competition solve time for a task; it is a difficulty reference, not the agent’s runtime.
Cybench official website and metric definitions · Stanford CRFM introduction to Cybench
When it is useful
Use it to study tool use, sustained problem solving, and the effect of explicit intermediate guidance. The distinction between partial progress and a completed challenge helps identify where an agent stalls, rather than treating every incomplete run as equally informative.
Compare matching protocols
CTF performance is evidence about these challenge environments, not a direct measurement of real enterprise intrusions. Published results may use subsets of the original suite or different trial aggregation. Compare the task list, guidance mode, agent harness, and budget before interpreting a percentage difference.
Cybench official website and metric definitions · Cybench paper