The published 1,507-task dataset, its information levels, and current submission rules.
What it measures
CyberGym focuses on vulnerability reproduction across 1,507 real vulnerabilities from 188 software projects. Its central question is whether an agent can understand an existing bug report, investigate the relevant code, and produce a test input that demonstrates the reported failure.
What the agent receives
The main Level 1 setting provides source code and a textual vulnerability description. Other levels change the available evidence: Level 0 removes the description, Level 2 adds a crash trace, and Level 3 adds patch information. These are different information settings, so the level belongs beside every result.
How success is checked
Verification checks that the submitted input triggers the vulnerability in the unpatched program and does not trigger it after the fix. Current submission rules require the agent to select one final proof of concept; success from any intermediate attempt is a different statistic and should be identified separately.
When it is useful
Use it to compare how agents read unfamiliar code and turn written reports into executable evidence. For an engineering comparison, consider success together with token use, elapsed time, and tool requests, which the submission guidance asks evaluators to report.
What a score leaves out
A successful reproduction does not establish complete system compromise. Agent scaffolding, tool budgets, network access, and access to runnable target environments affect comparability. The current default setup withholds patched runtime images from the agent; do not confuse that rule with the paper’s separate patch-informed information level.