The initial v8-bench release: 41 patched V8 vulnerabilities and 16 capability flags.
What it measures
ExploitBench’s first benchmark, v8-bench, uses 41 patched vulnerabilities in the V8 JavaScript and WebAssembly engine. It records 16 capability flags grouped into five tiers, ranging from reaching affected code and reproducing a bug to stronger control capabilities. The flags preserve useful evidence of partial progress.
A known-vulnerability setting
The agent receives vulnerability context, including a patch diff, source code, and relevant program builds, and produces a test artifact for the benchmark environment. This is evaluation on already identified, patched bugs. It should be distinguished from discovering an unknown vulnerability without such starting information.
ExploitBench official repository · Official V8 benchmark specification
Read capability coverage carefully
Deterministic verifiers check the capability flags; an LLM judge does not assign the result. Capability coverage measures achieved flags, not the percentage of vulnerabilities fully exploited. The public results distinguish fixed experiments from a maximum-achievement view with varying seeds and budgets, so preserve the reported aggregation and regime.
ExploitBench official methodology and result views · Official V8 benchmark specification
When it is useful
Use it to investigate where progress stops within a difficult software target and to compare agent configurations under controlled conditions. A tier profile can distinguish reliable bug reproduction from later capabilities that a single success percentage would hide, while still crediting intermediate work.
Version and protocol boundaries
The initial suite is specific to V8 and public patched bugs; it does not establish general zero-day discovery ability. Baseline runs, automated coaching, and native CLI agents use different protocols. Public prior information and runtime conditions can affect results, making matched settings essential for a fair comparison.
ExploitBench official methodology and result views · ExploitBench paper · ExploitBench official repository