Model reports

PersonaCodeBench

Can a coding agent finish the software and meet the user’s explicit working constraints? Joint pass requires both.

AI Benchmark Radar / Equality Machine · Reviewed 2026-11-05

Paper v1 published 3 November 2026; reviewed 5 November. The 100-task Lite leaderboard and opening-disclosure variant are primary; native-only ablations and conditional subsets are separate.

Correct software that fits the user

Lite pairs 100 coding tasks with personas and 186 added requirements. Obligations are fixed before evaluation and progressively disclosed; they cover implementation, permissions, verification and handoff.

PersonaCodeBench · original paper v1

A fixed denominator with two checks

Joint pass requires native correctness and persona acceptance. All 100 assignments remain in the denominator: errors and unexecuted tasks get no credit, so low yield includes coverage failures.

PersonaCodeBench · original paper v1

Retain effort and runtime scope

Source scaffolds use Harbor 0.23.0/Docker and GPT-6 Luna medium as simulator/judge. GLM-5.3 is textual-only; Kimi-K3 mixes repaired settings. Exact run dates are unknown.

PersonaCodeBench · original paper v1

Acceptance is not human satisfaction

Persona acceptance operationalizes explicit obligations. Conditional subsets exclude timeouts; their model-specific denominators cannot rank full-set performance. The paper-linked repository returned 404 during review.

PersonaCodeBench · original paper v1

Full and Lite composition

Original release counts.

Release composition
Source / quantity	Full	Lite
Terminal-bench 3	52	23
DeepSWE	64	28
SWE-bench-Live	45	19
SWE-Together	9	4
SWE-bench Pro	61	26
Total pairs	231	100
evolved requirements	422	186
Distinct persona ID strings	66	35

The development collection and the public Lite release have different scopes; primary reported results use the 100-task Lite release.

PersonaCodeBench · table-1

All reported Lite configurations

Original fixed-denominator results.

Joint pass by source
Model	Effort	Joint %	Terminal-bench 3 %	SWE-bench-Live %	DeepSWE %	SWE-bench Pro %	SWE-Together %
Gemini 4 Argon	medium	19.0	8.7	15.8	10.7	34.6	50.0
Gemini 4 Argon	high	23.0	13.0	26.3	14.3	34.6	50.0
Claude Haiku 5.5	high	11.0	0.0	36.8	0.0	11.5	25.0
GPT-6.1 Sol	medium	31.0	4.3	52.6	14.3	53.8	50.0
GPT-6.1 Sol	high	38.0	17.4	47.4	39.3	42.3	75.0
GPT-6.1 Sol	xhigh	42.0	17.4	31.6	57.1	57.7	25.0
Claude Sonnet 5.5	high	26.0	8.7	26.3	0.0	65.4	50.0
Claude Opus 5.5	medium	40.0	8.7	57.9	21.4	53.8	50.0
Claude Opus 5.5	high	39.0	4.3	36.8	35.7	53.8	50.0
GPT-6 Sol	max	18.0	0.0	31.6	0.0	46.2	0.0
GPT-6 Luna	medium	19.0	0.0	31.6	7.1	34.6	50.0
Grok 4.7	high	24.0	8.7	26.3	25.0	30.8	50.0
DeepSeek V4.1	max	5.0	0.0	0.0	0.0	19.2	0.0
GPT-6 Astra	medium	38.0	17.4	42.1	32.1	57.7	50.0
GPT-6 Astra	high	45.0	17.4	52.6	53.6	53.8	50.0
GPT-6 Astra	xhigh	45.0	8.7	47.4	42.9	73.1	75.0
GPT-6 Astra	max	48.0	17.4	36.8	53.6	80.8	25.0
Gemini 3.8 Flash	medium	17.0	0.0	36.8	0.0	30.8	50.0
Gemini 3.8 Flash	high	17.0	0.0	36.8	0.0	34.6	25.0
Claude Fable 5.1	high	39.0	4.3	36.8	39.3	69.2	50.0
Qwen 3.8 Flash Next	medium	1.0	0.0	0.0	0.0	3.8	0.0
GLM-5.3 Flash	max	15.0	0.0	0.0	0.0	50.0	50.0
Qwen 3.8 27B	medium	6.0	0.0	5.3	0.0	15.4	25.0
GLM-5.3 †	high	22.0	4.3	31.6	0.0	50.0	50.0
Grok 4.6	medium	17.0	0.0	0.0	3.6	53.8	50.0
Kimi-K3	mixed	6.0	0.0	0.0	0.0	15.4	50.0
GPT-5.6 Terra	xhigh	34.0	8.7	36.8	35.7	50.0	50.0
GPT-5.5	xhigh	18.0	4.3	10.5	21.4	26.9	50.0

All 100 tasks; source denominators are 23/19/28/26/4 in column order. † textual only. Kimi mixed settings.

PersonaCodeBench · table-2

GPT-6 Luna by persona

Counts accompany percentages.

Assigned-task outcomes
Persona	Terminal-bench 3	SWE-bench Pro	SWE-bench-Live	SWE-Together	DeepSWE
Student Requiring explanations	0.0% 0/4	20.0% 1/5	0.0% 0/5	0.0% 0/1	20.0% 1/5
First PR contributor	0.0% 0/1	33.3% 3/9	–	0.0% 0/1	0.0% 0/8
Outcome-oriented manager	0.0% 0/3	–	50.0% 1/2	–	7.7% 1/13
Test-focused lead	0.0% 0/2	66.7% 2/3	0.0% 0/1	–	–
Senior reviewer	0.0% 0/2	–	50.0% 1/2	100.0% 2/2	–
Analyst Without index permissions	–	25.0% 1/4	100.0% 2/2	–	–
Designer	0.0% 0/6	0.0% 0/2	0.0% 0/1	–	0.0% 0/1
Shop owner Unable to install software	0.0% 0/2	0.0% 0/1	0.0% 0/1	–	0.0% 0/1
Founder With leaked credentials	–	100.0% 2/2	50.0% 1/2	–	–
Data scientist	0.0% 0/3	–	33.3% 1/3	–	–
Total	0.0% 0/23	34.6% 9/26	31.6% 6/19	50.0% 2/4	7.1% 2/28

Dashes mean no assigned task; not a zero score.

PersonaCodeBench · table-3

Disclosure and persona ablations

Single-run comparisons; native-only acceptance changes the contract.

GPT-6 Luna ablations
Variant	Reported joint %	Completion %	Personalization %
PersonaCodeBench	19.0%	24.0%	50.0%
PersonaCodeBench w. Opening	20.0%	26.0%	51.0%
PersonaCodeBench w/o Persona	45.0%	45.0%	–

Without Persona retains neutral-user interaction but removes persona grading: its 45% is native-only success, excluded from the primary joint series.

PersonaCodeBench · table-4

Conditional non-timeout results

Selected common-task sets; not the 100-task leaderboard.

Values stated in Appendix F
Model	Effort / budget	Common tasks	Conditional %
Claude Fable 5.1	max / 1×	31	77.4
Claude Opus 5.5	max / 1×	7	85.7
GPT-6.1 Sol	medium / 1×	69	29.0
GPT-6.1 Sol	xhigh / 1×	69	46.4
GPT-6.1 Sol	max / 1×	69	40.6
GPT-6 Sol	high / 1×	19	31.6
GPT-6 Sol	max / 1×	19	63.2
GPT-6 Astra	max / 2×	72	50.0
GPT-6 Astra	max / 3×	72	52.8
GPT-6.1 Sol	max / 2×	76	50.0
GPT-6.1 Sol	max / 3×	76	46.1
GPT-6 Sol	max / 2×	20	60.0
GPT-6 Sol	max / 3×	20	55.0

Selection differs across models and can change from 1× to 2×/3×. Original Figure 11 contains further plotted settings.

PersonaCodeBench · conditional

Separate interaction and infrastructure outcomes

Later runtime phase limits are documented independently of source-specific inner limits.

Documented later-runtime watchdogs
Phase	Default limit	Exhaustion
Preparation	7,200 s	Infrastructure; score unavailable
Interaction	Task-specific; override supported	Failed attempt; available grades retained
Verification	max(900, verifier timeout + 600) s	Infrastructure; incomplete verification

Calibration/materialization precede these clocks. This does not establish the exact revision used for each leaderboard run.

PersonaCodeBench · timing

Original publication

Paper v1 published 3 November 2026; reviewed 5 November. The 100-task Lite leaderboard and opening-disclosure variant are primary; native-only ablations and conditional subsets are separate.

Reviewed 2026-11-05

Original paper · PersonaCodeBench · original paper

Evidence available for this benchmark

This source record contains 29 numeric observations. The source distinguishes 20 model identities or configurations. A model may appear more than once; check the version, tools and run budget before comparing. Read results and row-level evidence →

Reported metric: Joint pass over all 100 assigned Lite tasks
Recorded unit: percent
Metric direction: higher is better

Start with these resources

Catalog source · arxiv.org

Cited documents and dates (1)

PersonaCodeBench · original v1 evaluation · 2026-11-03

Evaluation context and evidence

Source: Model reports
Publisher: Not recorded
Benchmark release: 2026-11-03
Collected: Not recorded
Modality: Not recorded

These are source-reported observations, not independent reruns by this site. Check the task version, evaluation protocol, unit and reporting date before comparing models. Results from different setups are not a universal ranking.

Reported results 29

Shown in source observation order, not ranked. Dates retain the precision provided by the source.

Model	Value	Unit	Protocol	Reported date and basis	Source
GPT-6 Luna	20.0	percent	all evolved requirements disclosed at opening; acceptance, online gates and permissions retained; single-run Table 4 ablation; medium effort; 100 assigned Lite task-persona pairs; native completion AND all mandatory persona requirements; every non-pass including errors/unexecuted is zero; source-specific scaffold, Harbor 0.23.0/Docker; GPT-6 Luna medium user simulator and judge; source budget/runtime revision applies, exact per-run timing revision and execution dates unspecified	2026-11-03 · document_publication	PersonaCodeBench · original v1 evaluation
Claude Fable 5.1	39.0	percent	progressive disclosure; high effort; 100 assigned Lite task-persona pairs; native completion AND all mandatory persona requirements; every non-pass including errors/unexecuted is zero; source-specific scaffold, Harbor 0.23.0/Docker; GPT-6 Luna medium user simulator and judge; source budget/runtime revision applies, exact per-run timing revision and execution dates unspecified	2026-11-03 · document_publication	PersonaCodeBench · original v1 evaluation
Claude Haiku 5.5	11.0	percent	progressive disclosure; high effort; 100 assigned Lite task-persona pairs; native completion AND all mandatory persona requirements; every non-pass including errors/unexecuted is zero; source-specific scaffold, Harbor 0.23.0/Docker; GPT-6 Luna medium user simulator and judge; source budget/runtime revision applies, exact per-run timing revision and execution dates unspecified	2026-11-03 · document_publication	PersonaCodeBench · original v1 evaluation
Claude Opus 5.5	39.0	percent	progressive disclosure; high effort; 100 assigned Lite task-persona pairs; native completion AND all mandatory persona requirements; every non-pass including errors/unexecuted is zero; source-specific scaffold, Harbor 0.23.0/Docker; GPT-6 Luna medium user simulator and judge; source budget/runtime revision applies, exact per-run timing revision and execution dates unspecified	2026-11-03 · document_publication	PersonaCodeBench · original v1 evaluation
Claude Sonnet 5.5	26.0	percent	progressive disclosure; high effort; 100 assigned Lite task-persona pairs; native completion AND all mandatory persona requirements; every non-pass including errors/unexecuted is zero; source-specific scaffold, Harbor 0.23.0/Docker; GPT-6 Luna medium user simulator and judge; source budget/runtime revision applies, exact per-run timing revision and execution dates unspecified	2026-11-03 · document_publication	PersonaCodeBench · original v1 evaluation
GPT-6 Astra	45.0	percent	progressive disclosure; high effort; 100 assigned Lite task-persona pairs; native completion AND all mandatory persona requirements; every non-pass including errors/unexecuted is zero; source-specific scaffold, Harbor 0.23.0/Docker; GPT-6 Luna medium user simulator and judge; source budget/runtime revision applies, exact per-run timing revision and execution dates unspecified	2026-11-03 · document_publication	PersonaCodeBench · original v1 evaluation
GPT-6.1 Sol	38.0	percent	progressive disclosure; high effort; 100 assigned Lite task-persona pairs; native completion AND all mandatory persona requirements; every non-pass including errors/unexecuted is zero; source-specific scaffold, Harbor 0.23.0/Docker; GPT-6 Luna medium user simulator and judge; source budget/runtime revision applies, exact per-run timing revision and execution dates unspecified	2026-11-03 · document_publication	PersonaCodeBench · original v1 evaluation
Gemini 3.8 Flash	17.0	percent	progressive disclosure; high effort; 100 assigned Lite task-persona pairs; native completion AND all mandatory persona requirements; every non-pass including errors/unexecuted is zero; source-specific scaffold, Harbor 0.23.0/Docker; GPT-6 Luna medium user simulator and judge; source budget/runtime revision applies, exact per-run timing revision and execution dates unspecified	2026-11-03 · document_publication	PersonaCodeBench · original v1 evaluation
Gemini 4 Argon	23.0	percent	progressive disclosure; high effort; 100 assigned Lite task-persona pairs; native completion AND all mandatory persona requirements; every non-pass including errors/unexecuted is zero; source-specific scaffold, Harbor 0.23.0/Docker; GPT-6 Luna medium user simulator and judge; source budget/runtime revision applies, exact per-run timing revision and execution dates unspecified	2026-11-03 · document_publication	PersonaCodeBench · original v1 evaluation
Grok 4.7	24.0	percent	progressive disclosure; high effort; 100 assigned Lite task-persona pairs; native completion AND all mandatory persona requirements; every non-pass including errors/unexecuted is zero; source-specific scaffold, Harbor 0.23.0/Docker; GPT-6 Luna medium user simulator and judge; source budget/runtime revision applies, exact per-run timing revision and execution dates unspecified	2026-11-03 · document_publication	PersonaCodeBench · original v1 evaluation
GLM-5.3	22.0	percent	progressive disclosure; high effort; 100 assigned Lite task-persona pairs; native completion AND all mandatory persona requirements; every non-pass including errors/unexecuted is zero; source-specific scaffold, Harbor 0.23.0/Docker; GPT-6 Luna medium user simulator and judge; source budget/runtime revision applies, exact per-run timing revision and execution dates unspecified; textual-only grading (original dagger footnote)	2026-11-03 · document_publication	PersonaCodeBench · original v1 evaluation
DeepSeek V4.1	5.0	percent	progressive disclosure; max effort; 100 assigned Lite task-persona pairs; native completion AND all mandatory persona requirements; every non-pass including errors/unexecuted is zero; source-specific scaffold, Harbor 0.23.0/Docker; GPT-6 Luna medium user simulator and judge; source budget/runtime revision applies, exact per-run timing revision and execution dates unspecified	2026-11-03 · document_publication	PersonaCodeBench · original v1 evaluation
GLM-5.3 Flash	15.0	percent	progressive disclosure; max effort; 100 assigned Lite task-persona pairs; native completion AND all mandatory persona requirements; every non-pass including errors/unexecuted is zero; source-specific scaffold, Harbor 0.23.0/Docker; GPT-6 Luna medium user simulator and judge; source budget/runtime revision applies, exact per-run timing revision and execution dates unspecified	2026-11-03 · document_publication	PersonaCodeBench · original v1 evaluation
GPT-6 Astra	48.0	percent	progressive disclosure; max effort; 100 assigned Lite task-persona pairs; native completion AND all mandatory persona requirements; every non-pass including errors/unexecuted is zero; source-specific scaffold, Harbor 0.23.0/Docker; GPT-6 Luna medium user simulator and judge; source budget/runtime revision applies, exact per-run timing revision and execution dates unspecified	2026-11-03 · document_publication	PersonaCodeBench · original v1 evaluation
GPT-6 Sol	18.0	percent	progressive disclosure; max effort; 100 assigned Lite task-persona pairs; native completion AND all mandatory persona requirements; every non-pass including errors/unexecuted is zero; source-specific scaffold, Harbor 0.23.0/Docker; GPT-6 Luna medium user simulator and judge; source budget/runtime revision applies, exact per-run timing revision and execution dates unspecified	2026-11-03 · document_publication	PersonaCodeBench · original v1 evaluation
Claude Opus 5.5	40.0	percent	progressive disclosure; medium effort; 100 assigned Lite task-persona pairs; native completion AND all mandatory persona requirements; every non-pass including errors/unexecuted is zero; source-specific scaffold, Harbor 0.23.0/Docker; GPT-6 Luna medium user simulator and judge; source budget/runtime revision applies, exact per-run timing revision and execution dates unspecified	2026-11-03 · document_publication	PersonaCodeBench · original v1 evaluation
GPT-6 Astra	38.0	percent	progressive disclosure; medium effort; 100 assigned Lite task-persona pairs; native completion AND all mandatory persona requirements; every non-pass including errors/unexecuted is zero; source-specific scaffold, Harbor 0.23.0/Docker; GPT-6 Luna medium user simulator and judge; source budget/runtime revision applies, exact per-run timing revision and execution dates unspecified	2026-11-03 · document_publication	PersonaCodeBench · original v1 evaluation
GPT-6 Luna	19.0	percent	progressive disclosure; medium effort; 100 assigned Lite task-persona pairs; native completion AND all mandatory persona requirements; every non-pass including errors/unexecuted is zero; source-specific scaffold, Harbor 0.23.0/Docker; GPT-6 Luna medium user simulator and judge; source budget/runtime revision applies, exact per-run timing revision and execution dates unspecified	2026-11-03 · document_publication	PersonaCodeBench · original v1 evaluation
GPT-6.1 Sol	31.0	percent	progressive disclosure; medium effort; 100 assigned Lite task-persona pairs; native completion AND all mandatory persona requirements; every non-pass including errors/unexecuted is zero; source-specific scaffold, Harbor 0.23.0/Docker; GPT-6 Luna medium user simulator and judge; source budget/runtime revision applies, exact per-run timing revision and execution dates unspecified	2026-11-03 · document_publication	PersonaCodeBench · original v1 evaluation
Gemini 3.8 Flash	17.0	percent	progressive disclosure; medium effort; 100 assigned Lite task-persona pairs; native completion AND all mandatory persona requirements; every non-pass including errors/unexecuted is zero; source-specific scaffold, Harbor 0.23.0/Docker; GPT-6 Luna medium user simulator and judge; source budget/runtime revision applies, exact per-run timing revision and execution dates unspecified	2026-11-03 · document_publication	PersonaCodeBench · original v1 evaluation
Gemini 4 Argon	19.0	percent	progressive disclosure; medium effort; 100 assigned Lite task-persona pairs; native completion AND all mandatory persona requirements; every non-pass including errors/unexecuted is zero; source-specific scaffold, Harbor 0.23.0/Docker; GPT-6 Luna medium user simulator and judge; source budget/runtime revision applies, exact per-run timing revision and execution dates unspecified	2026-11-03 · document_publication	PersonaCodeBench · original v1 evaluation
Grok 4.6	17.0	percent	progressive disclosure; medium effort; 100 assigned Lite task-persona pairs; native completion AND all mandatory persona requirements; every non-pass including errors/unexecuted is zero; source-specific scaffold, Harbor 0.23.0/Docker; GPT-6 Luna medium user simulator and judge; source budget/runtime revision applies, exact per-run timing revision and execution dates unspecified	2026-11-03 · document_publication	PersonaCodeBench · original v1 evaluation
Qwen 3.8 27B	6.0	percent	progressive disclosure; medium effort; 100 assigned Lite task-persona pairs; native completion AND all mandatory persona requirements; every non-pass including errors/unexecuted is zero; source-specific scaffold, Harbor 0.23.0/Docker; GPT-6 Luna medium user simulator and judge; source budget/runtime revision applies, exact per-run timing revision and execution dates unspecified	2026-11-03 · document_publication	PersonaCodeBench · original v1 evaluation
Qwen 3.8 Flash Next	1.0	percent	progressive disclosure; medium effort; 100 assigned Lite task-persona pairs; native completion AND all mandatory persona requirements; every non-pass including errors/unexecuted is zero; source-specific scaffold, Harbor 0.23.0/Docker; GPT-6 Luna medium user simulator and judge; source budget/runtime revision applies, exact per-run timing revision and execution dates unspecified	2026-11-03 · document_publication	PersonaCodeBench · original v1 evaluation
Kimi-K3	6.0	percent	progressive disclosure; mixed effort; 100 assigned Lite task-persona pairs; native completion AND all mandatory persona requirements; every non-pass including errors/unexecuted is zero; source-specific scaffold, Harbor 0.23.0/Docker; GPT-6 Luna medium user simulator and judge; source budget/runtime revision applies, exact per-run timing revision and execution dates unspecified; mixed original settings and high-effort repairs	2026-11-03 · document_publication	PersonaCodeBench · original v1 evaluation
GPT-5.5	18.0	percent	progressive disclosure; xhigh effort; 100 assigned Lite task-persona pairs; native completion AND all mandatory persona requirements; every non-pass including errors/unexecuted is zero; source-specific scaffold, Harbor 0.23.0/Docker; GPT-6 Luna medium user simulator and judge; source budget/runtime revision applies, exact per-run timing revision and execution dates unspecified	2026-11-03 · document_publication	PersonaCodeBench · original v1 evaluation
GPT-5.6 Terra	34.0	percent	progressive disclosure; xhigh effort; 100 assigned Lite task-persona pairs; native completion AND all mandatory persona requirements; every non-pass including errors/unexecuted is zero; source-specific scaffold, Harbor 0.23.0/Docker; GPT-6 Luna medium user simulator and judge; source budget/runtime revision applies, exact per-run timing revision and execution dates unspecified	2026-11-03 · document_publication	PersonaCodeBench · original v1 evaluation
GPT-6 Astra	45.0	percent	progressive disclosure; xhigh effort; 100 assigned Lite task-persona pairs; native completion AND all mandatory persona requirements; every non-pass including errors/unexecuted is zero; source-specific scaffold, Harbor 0.23.0/Docker; GPT-6 Luna medium user simulator and judge; source budget/runtime revision applies, exact per-run timing revision and execution dates unspecified	2026-11-03 · document_publication	PersonaCodeBench · original v1 evaluation
GPT-6.1 Sol	42.0	percent	progressive disclosure; xhigh effort; 100 assigned Lite task-persona pairs; native completion AND all mandatory persona requirements; every non-pass including errors/unexecuted is zero; source-specific scaffold, Harbor 0.23.0/Docker; GPT-6 Luna medium user simulator and judge; source budget/runtime revision applies, exact per-run timing revision and execution dates unspecified	2026-11-03 · document_publication	PersonaCodeBench · original v1 evaluation

Correct software that fits the user

A fixed denominator with two checks

Retain effort and runtime scope

Acceptance is not human satisfaction

Full and Lite composition

All reported Lite configurations

GPT-6 Luna by persona

Disclosure and persona ablations

Conditional non-timeout results

Separate interaction and infrastructure outcomes

Original publication

Evidence available for this benchmark

Start with these resources

Evaluation context and evidence

Reported results 29

Explore related benchmarks

Put this benchmark in context

Continue exploring