Datacurve's DeepSWE v1 / v1.1. The task collection is shared, but verification changed in v1.1; check the version on each score.
What the agent actually does
Datacurve's collection contains 113 tasks across 91 repositories in five programming languages. An agent explores a pinned codebase and implements a requested change. For example, the HTTPX task asks for incremental JSON streaming with both synchronous and asynchronous interfaces.
Datacurve · DeepSWE introduction and methodology · Official task · HTTPX streaming JSON
How a solution passes
In v1.1, only the committed patch is applied in a fresh verifier container. Behavioral and designated regression tests decide success, rather than similarity to a reference implementation. Required tests must pass; partial test progress is diagnostic information, not partial task credit.
Datacurve · DeepSWE v1.1 verification changes · Official verifier · Required behavior and regression tests
Read pass@1 and pass@4 separately
The paper's pass@1 averages each task's success fraction across independent attempts. Its pass@4 asks whether any of up to four attempts succeeds. Compare the same benchmark version, agent framework and budget, and read uncertainty alongside small score differences.
When it is useful
Use the task definitions and execution traces to examine repository exploration, feature integration and missed requirements. The published mini-swe-agent setup gives models a shared bash interface; it does not reproduce every model's native coding product.
What a pass cannot establish
Binary tests cannot measure all aspects of maintainability or correctness. The paper treats its LLM-assisted verifier audit as a second opinion, not ground truth. Original tasks reduce exposure to existing public fixes, but publication can create future contamination risk.
Read verifier audits alongside the leaderboard
Independent reviewers have reported remaining verifier problems in v1.1, including conflicts between hidden tests and agent-authored tests. These are version-specific audit findings, not a recalculated model ranking. A low score warrants inspection of the failed tests as well as the proposed code.
Independent audit · DeepSWE v1.1 issue #52 · Samaritan Research · DeepSWE v1.1 benchmark review