The 20-task benchmark and 520-run experiment in arXiv v1. Each model–effort pair remains a separate configuration; pooled model summaries have different denominators.
Whole repositories, not isolated functions
Twenty pinned open-source repositories cover seven language rewrites, seven framework rewrites, three platform ports and three build-toolchain migrations. Each attempt receives 6–30 hours. The original APIs, artifacts and observable behavior must remain while the old stack leaves the delivered implementation.
SWE Refactor Bench paper v1 · protocol and validity analysis · Official tasks · budgets and corpus coverage
Three independent questions
First, a source audit judges whether migration occurred, with three samples voting per criterion. Next, all 130,118 frozen checks across the corpus must pass where applicable. Finally, six coding agents get one hour each to seek a test that passes on the original and fails on the submission, reproducible three times.
SWE Refactor Bench paper v1 · protocol and validity analysis · Official repository · scoring and pinned clients
Composite score is not acceptance rate
A task gets zero if the migration gate or any required fixed check fails. Otherwise it receives 40 points plus 10 for each verifier that finds no counterexample. The leaderboard averages this 0–100 composite over tasks. Acceptance is stricter: the task must survive all six verifiers.
SWE Refactor Bench paper v1 · protocol and validity analysis · Official repository · scoring and pinned clients
Every evaluated reasoning configuration
Table 3 reports all 26 configurations. Behavioral pass and cost are different measurements from the composite score; neither is substituted for it.
| Model | Harness | Effort | Behavioral pass (%) | Composite (/100) | API USD/task |
|---|---|---|---|---|---|
| Claude Opus 5 | Claude Code | low | 92.5 | 20.5 | 34.8 |
| Claude Opus 5 | Claude Code | medium | 92.7 | 28.5 | 38.4 |
| Claude Opus 5 | Claude Code | high | 92.8 | 34.5 | 55.7 |
| Claude Opus 5 | Claude Code | xhigh | 92.8 | 47.0 | 74.9 |
| Claude Opus 5 | Claude Code | max | 91.9 | 31.0 | 72.4 |
| GPT-5.6 Sol | Codex | none | 63.6 | 4.0 | 2.9 |
| GPT-5.6 Sol | Codex | low | 62.0 | 7.0 | 5.9 |
| GPT-5.6 Sol | Codex | medium | 70.8 | 6.5 | 6.0 |
| GPT-5.6 Sol | Codex | high | 74.0 | 19.0 | 7.7 |
| GPT-5.6 Sol | Codex | xhigh | 73.5 | 9.5 | 19.1 |
| GPT-5.6 Sol | Codex | max | 84.1 | 28.5 | 143.5 |
| Kimi K3 | Claude Code | max | 93.9 | 19.5 | 28.9 |
| Claude Sonnet 5 | Claude Code | low | 79.2 | 4.0 | 4.4 |
| Claude Sonnet 5 | Claude Code | medium | 73.9 | 15.0 | 11.9 |
| Claude Sonnet 5 | Claude Code | high | 85.6 | 6.0 | 24.6 |
| Claude Sonnet 5 | Claude Code | xhigh | 76.5 | 9.0 | 27.0 |
| Claude Sonnet 5 | Claude Code | max | 84.3 | 8.5 | 27.5 |
| GPT-5.6 Luna | Codex | none | 66.4 | 4.0 | 1.6 |
| GPT-5.6 Luna | Codex | low | 58.3 | 4.0 | 1.7 |
| GPT-5.6 Luna | Codex | medium | 66.6 | 0.0 | 1.7 |
| GPT-5.6 Luna | Codex | high | 75.6 | 4.0 | 1.8 |
| GPT-5.6 Luna | Codex | xhigh | 83.8 | 5.5 | 2.9 |
| GPT-5.6 Luna | Codex | max | 89.1 | 10.5 | 2.8 |
| Qwen 3.8 Max | Claude Code | max | 74.7 | 10.0 | 14.5 |
| DeepSeek V4 Flash | Claude Code | max | 90.7 | 7.0 | 4.3 |
| GLM 5.2 | Claude Code | max | 85.2 | 6.5 | 17.5 |
The mean behavioral-check rate is not the fraction of fully accepted migrations. GPT models use Codex; all others use Claude Code. Costs are those reported for these runs.
Paper v1 Table 3 · all 26 configurations · Official leaderboard · configuration and pooled summaries
Do not confuse pooled and best-configuration scores
The official site also pools every run of each model. Proprietary models have five or six effort levels, while each open-weight model has one. These summaries answer a different question from a best-configuration comparison.
| Model | Client | Configurations | Runs | Accepted | Composite (/100) |
|---|---|---|---|---|---|
| claude-opus-5 | Claude Code | 5 | 100 | 16 | 32.30 |
| gpt-5.6-sol | Codex CLI | 6 | 120 | 5 | 12.42 |
| claude-sonnet-5 | Claude Code | 5 | 100 | 3 | 8.50 |
| gpt-5.6-luna | Codex CLI | 6 | 120 | 0 | 4.67 |
| kimi-k3 | Claude Code | 1 | 20 | 2 | 19.50 |
| qwen3.8-max | Claude Code | 1 | 20 | 2 | 10.00 |
| dsv4-flash | Claude Code | 1 | 20 | 0 | 7.00 |
| glm-5.2 | Claude Code | 1 | 20 | 0 | 6.50 |
Different effort mixtures and denominators: do not treat these as the same protocol or as additional independent experiments.
Passing is bounded evidence
One attempt per task does not establish run-to-run reliability. The migration audit uses a model judge and can disagree with humans; the paper checks this explicitly. Six verifiers failing to find a difference within an hour is evidence about that panel and budget, not a proof of program equivalence.
SWE Refactor Bench paper v1 · protocol and validity analysis · Official leaderboard · configuration and pooled summaries