AI EVALUATION, IN CONTEXT.BY EQUALITY MACHINE
Benchmark Radar.Benchmarks. Model results. Original sources.
/

Model reports

SWE Refactor Bench

A migration can pass the original tests without replacing anything. This benchmark checks that the stack really changed, every recorded behavior survived, and independent agents cannot find a new counterexample.

Official projectOriginal paperOfficial repositoryOfficial projectCatalog source

AI Benchmark Radar / Equality Machine · Reviewed 2026-10-27

The 20-task benchmark and 520-run experiment in arXiv v1. Each model–effort pair remains a separate configuration; pooled model summaries have different denominators.

Whole repositories, not isolated functions

Twenty pinned open-source repositories cover seven language rewrites, seven framework rewrites, three platform ports and three build-toolchain migrations. Each attempt receives 6–30 hours. The original APIs, artifacts and observable behavior must remain while the old stack leaves the delivered implementation.

SWE Refactor Bench paper v1 · protocol and validity analysis · Official tasks · budgets and corpus coverage

Three independent questions

First, a source audit judges whether migration occurred, with three samples voting per criterion. Next, all 130,118 frozen checks across the corpus must pass where applicable. Finally, six coding agents get one hour each to seek a test that passes on the original and fails on the submission, reproducible three times.

SWE Refactor Bench paper v1 · protocol and validity analysis · Official repository · scoring and pinned clients

Composite score is not acceptance rate

A task gets zero if the migration gate or any required fixed check fails. Otherwise it receives 40 points plus 10 for each verifier that finds no counterexample. The leaderboard averages this 0–100 composite over tasks. Acceptance is stricter: the task must survive all six verifiers.

SWE Refactor Bench paper v1 · protocol and validity analysis · Official repository · scoring and pinned clients

Every evaluated reasoning configuration

Table 3 reports all 26 configurations. Behavioral pass and cost are different measurements from the composite score; neither is substituted for it.

Paper v1 Table 3 · one attempt on each of 20 tasks
ModelHarnessEffortBehavioral pass (%)Composite (/100)API USD/task
Claude Opus 5Claude Codelow92.520.534.8
Claude Opus 5Claude Codemedium92.728.538.4
Claude Opus 5Claude Codehigh92.834.555.7
Claude Opus 5Claude Codexhigh92.847.074.9
Claude Opus 5Claude Codemax91.931.072.4
GPT-5.6 SolCodexnone63.64.02.9
GPT-5.6 SolCodexlow62.07.05.9
GPT-5.6 SolCodexmedium70.86.56.0
GPT-5.6 SolCodexhigh74.019.07.7
GPT-5.6 SolCodexxhigh73.59.519.1
GPT-5.6 SolCodexmax84.128.5143.5
Kimi K3Claude Codemax93.919.528.9
Claude Sonnet 5Claude Codelow79.24.04.4
Claude Sonnet 5Claude Codemedium73.915.011.9
Claude Sonnet 5Claude Codehigh85.66.024.6
Claude Sonnet 5Claude Codexhigh76.59.027.0
Claude Sonnet 5Claude Codemax84.38.527.5
GPT-5.6 LunaCodexnone66.44.01.6
GPT-5.6 LunaCodexlow58.34.01.7
GPT-5.6 LunaCodexmedium66.60.01.7
GPT-5.6 LunaCodexhigh75.64.01.8
GPT-5.6 LunaCodexxhigh83.85.52.9
GPT-5.6 LunaCodexmax89.110.52.8
Qwen 3.8 MaxClaude Codemax74.710.014.5
DeepSeek V4 FlashClaude Codemax90.77.04.3
GLM 5.2Claude Codemax85.26.517.5

The mean behavioral-check rate is not the fraction of fully accepted migrations. GPT models use Codex; all others use Claude Code. Costs are those reported for these runs.

Paper v1 Table 3 · all 26 configurations · Official leaderboard · configuration and pooled summaries

Do not confuse pooled and best-configuration scores

The official site also pools every run of each model. Proprietary models have five or six effort levels, while each open-weight model has one. These summaries answer a different question from a best-configuration comparison.

Official pooled summaries · observed 2026-10-27
ModelClientConfigurationsRunsAcceptedComposite (/100)
claude-opus-5Claude Code51001632.30
gpt-5.6-solCodex CLI6120512.42
claude-sonnet-5Claude Code510038.50
gpt-5.6-lunaCodex CLI612004.67
kimi-k3Claude Code120219.50
qwen3.8-maxClaude Code120210.00
dsv4-flashClaude Code12007.00
glm-5.2Claude Code12006.50

Different effort mixtures and denominators: do not treat these as the same protocol or as additional independent experiments.

Official leaderboard · configuration and pooled summaries

Passing is bounded evidence

One attempt per task does not establish run-to-run reliability. The migration audit uses a model judge and can disagree with humans; the paper checks this explicitly. Six verifiers failing to find a difference within an hour is evidence about that panel and budget, not a proof of program equivalence.

SWE Refactor Bench paper v1 · protocol and validity analysis · Official leaderboard · configuration and pooled summaries

Original publication

The 20-task benchmark and 520-run experiment in arXiv v1. Each model–effort pair remains a separate configuration; pooled model summaries have different denominators.

Reviewed 2026-10-27

Evidence available for this benchmark

This source record contains 26 numeric observations. The source distinguishes 8 model identities or configurations. A model may appear more than once; check the version, tools and run budget before comparing. Read results and row-level evidence →

Reported metric
Composite score
Recorded unit
points
Metric direction
higher is better

Start with these resources

Cited documents and dates (1)

Recorded attention

Observed on: 2026-10-13

GitHub stars
53
Hugging Face paper upvotes
15
Star change · 30 days before observation
+4

View source snapshot

Dated attention observations, not live counts or benchmark scores.

Evaluation context and evidence

Source
Model reports
Publisher
Not recorded
Benchmark release
2026-08-24
Collected
Not recorded
Modality
Not recorded

These are source-reported observations, not independent reruns by this site. Check the task version, evaluation protocol, unit and reporting date before comparing models. Results from different setups are not a universal ranking.

Reported results 26

Shown in source observation order, not ranked. Dates retain the precision provided by the source.

ModelValueUnitProtocolReported date and basisSource
Claude Opus 534.5pointsClaude Code 2.1.220; reasoning effort high; one independent attempt per each of 20 tasks; per-task 6–30 hour wall-clock budget; no network except model endpoint; migration audit majority of 3 judge samples; all fixed tests required; six independent 1-hour differential verifiers; mean composite out of 1002026-08-24 · document_publicationSWE Refactor Bench paper v1 · Table 3
Claude Sonnet 56.0pointsClaude Code 2.1.220; reasoning effort high; one independent attempt per each of 20 tasks; per-task 6–30 hour wall-clock budget; no network except model endpoint; migration audit majority of 3 judge samples; all fixed tests required; six independent 1-hour differential verifiers; mean composite out of 1002026-08-24 · document_publicationSWE Refactor Bench paper v1 · Table 3
Claude Opus 520.5pointsClaude Code 2.1.220; reasoning effort low; one independent attempt per each of 20 tasks; per-task 6–30 hour wall-clock budget; no network except model endpoint; migration audit majority of 3 judge samples; all fixed tests required; six independent 1-hour differential verifiers; mean composite out of 1002026-08-24 · document_publicationSWE Refactor Bench paper v1 · Table 3
Claude Sonnet 54.0pointsClaude Code 2.1.220; reasoning effort low; one independent attempt per each of 20 tasks; per-task 6–30 hour wall-clock budget; no network except model endpoint; migration audit majority of 3 judge samples; all fixed tests required; six independent 1-hour differential verifiers; mean composite out of 1002026-08-24 · document_publicationSWE Refactor Bench paper v1 · Table 3
Claude Opus 531.0pointsClaude Code 2.1.220; reasoning effort max; one independent attempt per each of 20 tasks; per-task 6–30 hour wall-clock budget; no network except model endpoint; migration audit majority of 3 judge samples; all fixed tests required; six independent 1-hour differential verifiers; mean composite out of 1002026-08-24 · document_publicationSWE Refactor Bench paper v1 · Table 3
Claude Sonnet 58.5pointsClaude Code 2.1.220; reasoning effort max; one independent attempt per each of 20 tasks; per-task 6–30 hour wall-clock budget; no network except model endpoint; migration audit majority of 3 judge samples; all fixed tests required; six independent 1-hour differential verifiers; mean composite out of 1002026-08-24 · document_publicationSWE Refactor Bench paper v1 · Table 3
DeepSeek V4 Flash7.0pointsClaude Code 2.1.220; reasoning effort max; one independent attempt per each of 20 tasks; per-task 6–30 hour wall-clock budget; no network except model endpoint; migration audit majority of 3 judge samples; all fixed tests required; six independent 1-hour differential verifiers; mean composite out of 1002026-08-24 · document_publicationSWE Refactor Bench paper v1 · Table 3
GLM 5.26.5pointsClaude Code 2.1.220; reasoning effort max; one independent attempt per each of 20 tasks; per-task 6–30 hour wall-clock budget; no network except model endpoint; migration audit majority of 3 judge samples; all fixed tests required; six independent 1-hour differential verifiers; mean composite out of 1002026-08-24 · document_publicationSWE Refactor Bench paper v1 · Table 3
Kimi K319.5pointsClaude Code 2.1.220; reasoning effort max; one independent attempt per each of 20 tasks; per-task 6–30 hour wall-clock budget; no network except model endpoint; migration audit majority of 3 judge samples; all fixed tests required; six independent 1-hour differential verifiers; mean composite out of 1002026-08-24 · document_publicationSWE Refactor Bench paper v1 · Table 3
Qwen 3.8 Max10.0pointsClaude Code 2.1.220; reasoning effort max; one independent attempt per each of 20 tasks; per-task 6–30 hour wall-clock budget; no network except model endpoint; migration audit majority of 3 judge samples; all fixed tests required; six independent 1-hour differential verifiers; mean composite out of 1002026-08-24 · document_publicationSWE Refactor Bench paper v1 · Table 3
Claude Opus 528.5pointsClaude Code 2.1.220; reasoning effort medium; one independent attempt per each of 20 tasks; per-task 6–30 hour wall-clock budget; no network except model endpoint; migration audit majority of 3 judge samples; all fixed tests required; six independent 1-hour differential verifiers; mean composite out of 1002026-08-24 · document_publicationSWE Refactor Bench paper v1 · Table 3
Claude Sonnet 515.0pointsClaude Code 2.1.220; reasoning effort medium; one independent attempt per each of 20 tasks; per-task 6–30 hour wall-clock budget; no network except model endpoint; migration audit majority of 3 judge samples; all fixed tests required; six independent 1-hour differential verifiers; mean composite out of 1002026-08-24 · document_publicationSWE Refactor Bench paper v1 · Table 3
Claude Opus 547.0pointsClaude Code 2.1.220; reasoning effort xhigh; one independent attempt per each of 20 tasks; per-task 6–30 hour wall-clock budget; no network except model endpoint; migration audit majority of 3 judge samples; all fixed tests required; six independent 1-hour differential verifiers; mean composite out of 1002026-08-24 · document_publicationSWE Refactor Bench paper v1 · Table 3
Claude Sonnet 59.0pointsClaude Code 2.1.220; reasoning effort xhigh; one independent attempt per each of 20 tasks; per-task 6–30 hour wall-clock budget; no network except model endpoint; migration audit majority of 3 judge samples; all fixed tests required; six independent 1-hour differential verifiers; mean composite out of 1002026-08-24 · document_publicationSWE Refactor Bench paper v1 · Table 3
GPT-5.6 Luna4.0pointsCodex CLI 0.146.0; reasoning effort high; one independent attempt per each of 20 tasks; per-task 6–30 hour wall-clock budget; no network except model endpoint; migration audit majority of 3 judge samples; all fixed tests required; six independent 1-hour differential verifiers; mean composite out of 1002026-08-24 · document_publicationSWE Refactor Bench paper v1 · Table 3
GPT-5.6 Sol19.0pointsCodex CLI 0.146.0; reasoning effort high; one independent attempt per each of 20 tasks; per-task 6–30 hour wall-clock budget; no network except model endpoint; migration audit majority of 3 judge samples; all fixed tests required; six independent 1-hour differential verifiers; mean composite out of 1002026-08-24 · document_publicationSWE Refactor Bench paper v1 · Table 3
GPT-5.6 Luna4.0pointsCodex CLI 0.146.0; reasoning effort low; one independent attempt per each of 20 tasks; per-task 6–30 hour wall-clock budget; no network except model endpoint; migration audit majority of 3 judge samples; all fixed tests required; six independent 1-hour differential verifiers; mean composite out of 1002026-08-24 · document_publicationSWE Refactor Bench paper v1 · Table 3
GPT-5.6 Sol7.0pointsCodex CLI 0.146.0; reasoning effort low; one independent attempt per each of 20 tasks; per-task 6–30 hour wall-clock budget; no network except model endpoint; migration audit majority of 3 judge samples; all fixed tests required; six independent 1-hour differential verifiers; mean composite out of 1002026-08-24 · document_publicationSWE Refactor Bench paper v1 · Table 3
GPT-5.6 Luna10.5pointsCodex CLI 0.146.0; reasoning effort max; one independent attempt per each of 20 tasks; per-task 6–30 hour wall-clock budget; no network except model endpoint; migration audit majority of 3 judge samples; all fixed tests required; six independent 1-hour differential verifiers; mean composite out of 1002026-08-24 · document_publicationSWE Refactor Bench paper v1 · Table 3
GPT-5.6 Sol28.5pointsCodex CLI 0.146.0; reasoning effort max; one independent attempt per each of 20 tasks; per-task 6–30 hour wall-clock budget; no network except model endpoint; migration audit majority of 3 judge samples; all fixed tests required; six independent 1-hour differential verifiers; mean composite out of 1002026-08-24 · document_publicationSWE Refactor Bench paper v1 · Table 3
GPT-5.6 Luna0.0pointsCodex CLI 0.146.0; reasoning effort medium; one independent attempt per each of 20 tasks; per-task 6–30 hour wall-clock budget; no network except model endpoint; migration audit majority of 3 judge samples; all fixed tests required; six independent 1-hour differential verifiers; mean composite out of 1002026-08-24 · document_publicationSWE Refactor Bench paper v1 · Table 3
GPT-5.6 Sol6.5pointsCodex CLI 0.146.0; reasoning effort medium; one independent attempt per each of 20 tasks; per-task 6–30 hour wall-clock budget; no network except model endpoint; migration audit majority of 3 judge samples; all fixed tests required; six independent 1-hour differential verifiers; mean composite out of 1002026-08-24 · document_publicationSWE Refactor Bench paper v1 · Table 3
GPT-5.6 Luna4.0pointsCodex CLI 0.146.0; reasoning effort none; one independent attempt per each of 20 tasks; per-task 6–30 hour wall-clock budget; no network except model endpoint; migration audit majority of 3 judge samples; all fixed tests required; six independent 1-hour differential verifiers; mean composite out of 1002026-08-24 · document_publicationSWE Refactor Bench paper v1 · Table 3
GPT-5.6 Sol4.0pointsCodex CLI 0.146.0; reasoning effort none; one independent attempt per each of 20 tasks; per-task 6–30 hour wall-clock budget; no network except model endpoint; migration audit majority of 3 judge samples; all fixed tests required; six independent 1-hour differential verifiers; mean composite out of 1002026-08-24 · document_publicationSWE Refactor Bench paper v1 · Table 3
GPT-5.6 Luna5.5pointsCodex CLI 0.146.0; reasoning effort xhigh; one independent attempt per each of 20 tasks; per-task 6–30 hour wall-clock budget; no network except model endpoint; migration audit majority of 3 judge samples; all fixed tests required; six independent 1-hour differential verifiers; mean composite out of 1002026-08-24 · document_publicationSWE Refactor Bench paper v1 · Table 3
GPT-5.6 Sol9.5pointsCodex CLI 0.146.0; reasoning effort xhigh; one independent attempt per each of 20 tasks; per-task 6–30 hour wall-clock budget; no network except model endpoint; migration audit majority of 3 judge samples; all fixed tests required; six independent 1-hour differential verifiers; mean composite out of 1002026-08-24 · document_publicationSWE Refactor Bench paper v1 · Table 3

Explore related benchmarks

Suggestions based on shared evidence or source categories, not verified equivalents.

LiveCodeBenchShared source categories →AetherCodeShared source categories →ProgramBenchShared source categories →Aider PolyglotShared source categories →LiveCodeBenchShared source categories →Terminal-Bench v2.1Shared source categories →MBPPShared source categories →Web BenchShared source categories →

Put this benchmark in context

Coding benchmarksTools and agents benchmarks

Continue exploring

Search the catalog using this record’s source labels. Search matches are not verified equivalents.

coding →Model reports · all records →