| Claude Opus 5 | 34.5 | points | Claude Code 2.1.220; reasoning effort high; one independent attempt per each of 20 tasks; per-task 6–30 hour wall-clock budget; no network except model endpoint; migration audit majority of 3 judge samples; all fixed tests required; six independent 1-hour differential verifiers; mean composite out of 100 | 2026-08-24 · document_publication | SWE Refactor Bench paper v1 · Table 3 |
| Claude Sonnet 5 | 6.0 | points | Claude Code 2.1.220; reasoning effort high; one independent attempt per each of 20 tasks; per-task 6–30 hour wall-clock budget; no network except model endpoint; migration audit majority of 3 judge samples; all fixed tests required; six independent 1-hour differential verifiers; mean composite out of 100 | 2026-08-24 · document_publication | SWE Refactor Bench paper v1 · Table 3 |
| Claude Opus 5 | 20.5 | points | Claude Code 2.1.220; reasoning effort low; one independent attempt per each of 20 tasks; per-task 6–30 hour wall-clock budget; no network except model endpoint; migration audit majority of 3 judge samples; all fixed tests required; six independent 1-hour differential verifiers; mean composite out of 100 | 2026-08-24 · document_publication | SWE Refactor Bench paper v1 · Table 3 |
| Claude Sonnet 5 | 4.0 | points | Claude Code 2.1.220; reasoning effort low; one independent attempt per each of 20 tasks; per-task 6–30 hour wall-clock budget; no network except model endpoint; migration audit majority of 3 judge samples; all fixed tests required; six independent 1-hour differential verifiers; mean composite out of 100 | 2026-08-24 · document_publication | SWE Refactor Bench paper v1 · Table 3 |
| Claude Opus 5 | 31.0 | points | Claude Code 2.1.220; reasoning effort max; one independent attempt per each of 20 tasks; per-task 6–30 hour wall-clock budget; no network except model endpoint; migration audit majority of 3 judge samples; all fixed tests required; six independent 1-hour differential verifiers; mean composite out of 100 | 2026-08-24 · document_publication | SWE Refactor Bench paper v1 · Table 3 |
| Claude Sonnet 5 | 8.5 | points | Claude Code 2.1.220; reasoning effort max; one independent attempt per each of 20 tasks; per-task 6–30 hour wall-clock budget; no network except model endpoint; migration audit majority of 3 judge samples; all fixed tests required; six independent 1-hour differential verifiers; mean composite out of 100 | 2026-08-24 · document_publication | SWE Refactor Bench paper v1 · Table 3 |
| DeepSeek V4 Flash | 7.0 | points | Claude Code 2.1.220; reasoning effort max; one independent attempt per each of 20 tasks; per-task 6–30 hour wall-clock budget; no network except model endpoint; migration audit majority of 3 judge samples; all fixed tests required; six independent 1-hour differential verifiers; mean composite out of 100 | 2026-08-24 · document_publication | SWE Refactor Bench paper v1 · Table 3 |
| GLM 5.2 | 6.5 | points | Claude Code 2.1.220; reasoning effort max; one independent attempt per each of 20 tasks; per-task 6–30 hour wall-clock budget; no network except model endpoint; migration audit majority of 3 judge samples; all fixed tests required; six independent 1-hour differential verifiers; mean composite out of 100 | 2026-08-24 · document_publication | SWE Refactor Bench paper v1 · Table 3 |
| Kimi K3 | 19.5 | points | Claude Code 2.1.220; reasoning effort max; one independent attempt per each of 20 tasks; per-task 6–30 hour wall-clock budget; no network except model endpoint; migration audit majority of 3 judge samples; all fixed tests required; six independent 1-hour differential verifiers; mean composite out of 100 | 2026-08-24 · document_publication | SWE Refactor Bench paper v1 · Table 3 |
| Qwen 3.8 Max | 10.0 | points | Claude Code 2.1.220; reasoning effort max; one independent attempt per each of 20 tasks; per-task 6–30 hour wall-clock budget; no network except model endpoint; migration audit majority of 3 judge samples; all fixed tests required; six independent 1-hour differential verifiers; mean composite out of 100 | 2026-08-24 · document_publication | SWE Refactor Bench paper v1 · Table 3 |
| Claude Opus 5 | 28.5 | points | Claude Code 2.1.220; reasoning effort medium; one independent attempt per each of 20 tasks; per-task 6–30 hour wall-clock budget; no network except model endpoint; migration audit majority of 3 judge samples; all fixed tests required; six independent 1-hour differential verifiers; mean composite out of 100 | 2026-08-24 · document_publication | SWE Refactor Bench paper v1 · Table 3 |
| Claude Sonnet 5 | 15.0 | points | Claude Code 2.1.220; reasoning effort medium; one independent attempt per each of 20 tasks; per-task 6–30 hour wall-clock budget; no network except model endpoint; migration audit majority of 3 judge samples; all fixed tests required; six independent 1-hour differential verifiers; mean composite out of 100 | 2026-08-24 · document_publication | SWE Refactor Bench paper v1 · Table 3 |
| Claude Opus 5 | 47.0 | points | Claude Code 2.1.220; reasoning effort xhigh; one independent attempt per each of 20 tasks; per-task 6–30 hour wall-clock budget; no network except model endpoint; migration audit majority of 3 judge samples; all fixed tests required; six independent 1-hour differential verifiers; mean composite out of 100 | 2026-08-24 · document_publication | SWE Refactor Bench paper v1 · Table 3 |
| Claude Sonnet 5 | 9.0 | points | Claude Code 2.1.220; reasoning effort xhigh; one independent attempt per each of 20 tasks; per-task 6–30 hour wall-clock budget; no network except model endpoint; migration audit majority of 3 judge samples; all fixed tests required; six independent 1-hour differential verifiers; mean composite out of 100 | 2026-08-24 · document_publication | SWE Refactor Bench paper v1 · Table 3 |
| GPT-5.6 Luna | 4.0 | points | Codex CLI 0.146.0; reasoning effort high; one independent attempt per each of 20 tasks; per-task 6–30 hour wall-clock budget; no network except model endpoint; migration audit majority of 3 judge samples; all fixed tests required; six independent 1-hour differential verifiers; mean composite out of 100 | 2026-08-24 · document_publication | SWE Refactor Bench paper v1 · Table 3 |
| GPT-5.6 Sol | 19.0 | points | Codex CLI 0.146.0; reasoning effort high; one independent attempt per each of 20 tasks; per-task 6–30 hour wall-clock budget; no network except model endpoint; migration audit majority of 3 judge samples; all fixed tests required; six independent 1-hour differential verifiers; mean composite out of 100 | 2026-08-24 · document_publication | SWE Refactor Bench paper v1 · Table 3 |
| GPT-5.6 Luna | 4.0 | points | Codex CLI 0.146.0; reasoning effort low; one independent attempt per each of 20 tasks; per-task 6–30 hour wall-clock budget; no network except model endpoint; migration audit majority of 3 judge samples; all fixed tests required; six independent 1-hour differential verifiers; mean composite out of 100 | 2026-08-24 · document_publication | SWE Refactor Bench paper v1 · Table 3 |
| GPT-5.6 Sol | 7.0 | points | Codex CLI 0.146.0; reasoning effort low; one independent attempt per each of 20 tasks; per-task 6–30 hour wall-clock budget; no network except model endpoint; migration audit majority of 3 judge samples; all fixed tests required; six independent 1-hour differential verifiers; mean composite out of 100 | 2026-08-24 · document_publication | SWE Refactor Bench paper v1 · Table 3 |
| GPT-5.6 Luna | 10.5 | points | Codex CLI 0.146.0; reasoning effort max; one independent attempt per each of 20 tasks; per-task 6–30 hour wall-clock budget; no network except model endpoint; migration audit majority of 3 judge samples; all fixed tests required; six independent 1-hour differential verifiers; mean composite out of 100 | 2026-08-24 · document_publication | SWE Refactor Bench paper v1 · Table 3 |
| GPT-5.6 Sol | 28.5 | points | Codex CLI 0.146.0; reasoning effort max; one independent attempt per each of 20 tasks; per-task 6–30 hour wall-clock budget; no network except model endpoint; migration audit majority of 3 judge samples; all fixed tests required; six independent 1-hour differential verifiers; mean composite out of 100 | 2026-08-24 · document_publication | SWE Refactor Bench paper v1 · Table 3 |
| GPT-5.6 Luna | 0.0 | points | Codex CLI 0.146.0; reasoning effort medium; one independent attempt per each of 20 tasks; per-task 6–30 hour wall-clock budget; no network except model endpoint; migration audit majority of 3 judge samples; all fixed tests required; six independent 1-hour differential verifiers; mean composite out of 100 | 2026-08-24 · document_publication | SWE Refactor Bench paper v1 · Table 3 |
| GPT-5.6 Sol | 6.5 | points | Codex CLI 0.146.0; reasoning effort medium; one independent attempt per each of 20 tasks; per-task 6–30 hour wall-clock budget; no network except model endpoint; migration audit majority of 3 judge samples; all fixed tests required; six independent 1-hour differential verifiers; mean composite out of 100 | 2026-08-24 · document_publication | SWE Refactor Bench paper v1 · Table 3 |
| GPT-5.6 Luna | 4.0 | points | Codex CLI 0.146.0; reasoning effort none; one independent attempt per each of 20 tasks; per-task 6–30 hour wall-clock budget; no network except model endpoint; migration audit majority of 3 judge samples; all fixed tests required; six independent 1-hour differential verifiers; mean composite out of 100 | 2026-08-24 · document_publication | SWE Refactor Bench paper v1 · Table 3 |
| GPT-5.6 Sol | 4.0 | points | Codex CLI 0.146.0; reasoning effort none; one independent attempt per each of 20 tasks; per-task 6–30 hour wall-clock budget; no network except model endpoint; migration audit majority of 3 judge samples; all fixed tests required; six independent 1-hour differential verifiers; mean composite out of 100 | 2026-08-24 · document_publication | SWE Refactor Bench paper v1 · Table 3 |
| GPT-5.6 Luna | 5.5 | points | Codex CLI 0.146.0; reasoning effort xhigh; one independent attempt per each of 20 tasks; per-task 6–30 hour wall-clock budget; no network except model endpoint; migration audit majority of 3 judge samples; all fixed tests required; six independent 1-hour differential verifiers; mean composite out of 100 | 2026-08-24 · document_publication | SWE Refactor Bench paper v1 · Table 3 |
| GPT-5.6 Sol | 9.5 | points | Codex CLI 0.146.0; reasoning effort xhigh; one independent attempt per each of 20 tasks; per-task 6–30 hour wall-clock budget; no network except model endpoint; migration audit majority of 3 judge samples; all fixed tests required; six independent 1-hour differential verifiers; mean composite out of 100 | 2026-08-24 · document_publication | SWE Refactor Bench paper v1 · Table 3 |