Original GameASG-Bench paper v1 and the pinned public implementation. All tables retain the authors’ results and experimental settings; AI Benchmark Radar has not rerun them.
The deliverable: a complete browser game
Each of the 47 tasks asks for one executable index.html, covering 12 genres with 32 2D and 15 3D games. Agents receive target.md, game-spec.md and tdd.md. First-party code must fit inside the HTML; only task-permitted CDN libraries may remain external. The evaluator serves no companion assets.
GameASG-Bench · Task format and execution guide (pinned release)
A test interface tied to real gameplay
The window.__gameTest contract exposes reset, loadScenario, input and getSnapshot. A prepared scenario establishes a legal starting point; a player action must still cause the outcome. The interface must control the visible game itself, rather than a separate state that merely satisfies tests.
GameASG-Bench · Armor Alley public gameplay contract (pinned release)
Evidence layers and requirement priorities
L1 inspects the delivered source. L2 executes the game in Chromium, preparing a scenario, acting and observing the outcome on a fresh page for each check. P0 covers launch and interface prerequisites; P1 covers required gameplay; P2 covers extensions. Layer and priority answer different questions.
GameASG-Bench · Task format and execution guide (pinned release) · GameASG-Bench · Browser check isolation and execution code
Read task success and check rates separately
Strict success requires verified delivery, completed evaluation, every L1 check, and every applicable L2 P0/P1 check; P2 is excluded. Its denominator includes all 47 planned tasks, including failures and missing results. L1 and L2 Mean average task-level rates; priority columns pool applicable checks. NOT_APPLICABLE checks are excluded, including P1 when marked that way.
GameASG-Bench · Strict success, aggregation and cost estimation code · GameASG-Bench · Applicable-check and priority aggregation code
Nine model–harness configurations
All nine configurations complete evaluation on every task at maximum reasoning effort. Each task is attempted once. These are the authors’ observations, with no repeated-run uncertainty estimate.
| Model | Harness | Strict success: count/47 (%) | L1 % | L2 Mean % | L2 P0 % | L2 P1 % | L2 P2 % |
|---|---|---|---|---|---|---|---|
| GPT-6-Astra | Codex CLI | 26/47 (55.3) | 98.0 | 93.2 | 99.0 | 92.7 | 92.4 |
| Claude-Opus-5 | Claude Code | 24/47 (51.1) | 99.6 | 90.4 | 91.2 | 89.9 | 91.1 |
| GPT-5.6-Sol | Codex CLI | 21/47 (44.7) | 98.9 | 91.3 | 100.0 | 89.7 | 91.9 |
| DeepSeek-V4-Flash | Claude Code | 18/47 (38.3) | 99.4 | 89.3 | 98.0 | 86.7 | 90.2 |
| DeepSeek-V4-Pro | Claude Code | 15/47 (31.9) | 99.6 | 85.2 | 85.3 | 83.9 | 88.9 |
| Kimi-K3 | Claude Code | 15/47 (31.9) | 97.7 | 88.5 | 98.0 | 86.3 | 89.7 |
| GLM-5.2 | Claude Code | 11/47 (23.4) | 98.8 | 84.6 | 91.2 | 83.3 | 83.0 |
| Hunyuan-3 | Claude Code | 10/47 (21.3) | 98.3 | 81.7 | 96.1 | 78.2 | 82.3 |
| MiniMax-M3 | Claude Code | 7/47 (14.9) | 98.3 | 70.6 | 91.2 | 64.5 | 75.4 |
Parentheses contain percentages. A high mean check rate does not imply that a task passed every required check. Original model names, values and precision are retained; these are not reruns by this site.
GameASG-Bench paper v1 · Experimental settings and Tables 3–9 · GameASG-Bench · Strict success, aggregation and cost estimation code
Artifact size and resource use
Resource means use available measurements; missing entries are excluded rather than treated as zero. Input tokens combine input, cache reads and cache writes. Monetary values are estimates from the report script’s fixed rates, excluding request-specific surcharges and tool fees.
| Model | Harness | Artifact KB | Input tokens k | Output tokens k | Estimated cost USD |
|---|---|---|---|---|---|
| GPT-6-Astra | Codex CLI | 78.82 | 896.1 | 55.5 | 4.4445 |
| Claude-Opus-5 | Claude Code | 68.52 | 12,632.6 | 142.4 | 11.8500 |
| GPT-5.6-Sol | Codex CLI | 64.67 | 1,382.5 | 45.7 | 1.7867 |
| DeepSeek-V4-Flash | Claude Code | 85.21 | 13,038.4 | 203.6 | 0.2461 |
| DeepSeek-V4-Pro | Claude Code | 90.34 | 12,944.3 | 202.9 | 0.7431 |
| Kimi-K3 | Claude Code | 57.51 | 1,263.9 | 72.3 | 2.5518 |
| GLM-5.2 | Claude Code | 60.33 | 1,067.1 | 47.0 | 0.6184 |
| Hunyuan-3 | Claude Code | 52.88 | 2,474.5 | 88.9 | 0.1779 |
| MiniMax-M3 | Claude Code | 81.76 | 4,918.9 | 71.1 | 0.4069 |
Artifact sizes cover all 47 tasks per configuration. k = 1,000 tokens; KB = 1,024 bytes. These historical cost estimates are not current provider prices, and different accounting conventions limit efficiency comparisons.
GameASG-Bench paper v1 · Experimental settings and Tables 3–9 · GameASG-Bench · Strict success, aggregation and cost estimation code
Tool access changes the workflow
DeepSeek-V4-Flash is tested at maximum reasoning effort. The full-tool baseline uses Claude Code and a nominal 120-turn budget; the no-tool condition instead sends serialized API input. Syntax-only access permits node --check, without a browser.
| Tool setting | Strict success: count/47 (%) | L1 % | L2 P0 % | L2 P1 % | L2 P2 % |
|---|---|---|---|---|---|
| No tools (serialized API) | 7/47 (14.9) | 98.5 | 83.3 | 65.3 | 80.1 |
| File read/write only | 6/47 (12.8) | 98.8 | 87.3 | 66.2 | 73.4 |
| Files + syntax checking | 9/47 (19.1) | 98.2 | 86.3 | 63.4 | 69.3 |
| Full tool access | 18/47 (38.3) | 99.4 | 98.0 | 86.7 | 90.2 |
Every condition completes evaluation on 47 tasks. No tools = serialized API; file-only = read/write; syntax = files plus syntax checking; full = full tool access. The no-tool comparison also changes input delivery, so it does not isolate one tool alone.
GameASG-Bench paper v1 · Experimental settings and Tables 3–9
The denominator matters when runs stop early
This comparison fixes DeepSeek-V4-Flash, Claude Code, full tools and maximum reasoning effort. The conditional success column describes only tasks that reach evaluation; the all-planned column measures delivery across the full benchmark.
| Turns | Evaluated / planned | Success / all planned (%) | Success / evaluated (%) | L1 % | L2 P0 % | L2 P1 % | L2 P2 % |
|---|---|---|---|---|---|---|---|
| 30 | 10/47 | 7/47 (14.9) | 7/10 (70.0) | 100.0 | 100.0 | 94.0 | 97.6 |
| 60 | 32/47 | 13/47 (27.7) | 13/32 (40.6) | 99.7 | 88.6 | 81.6 | 88.5 |
| 120 | 47/47 | 18/47 (38.3) | 18/47 (38.3) | 99.4 | 98.0 | 86.7 | 90.2 |
The 30-turn result of 70.0% is 7 successes among 10 evaluated tasks; the primary result is 7/47 (14.9%). Check rates also describe the evaluated subset. Do not compare conditional rates as if all runs had completed.
GameASG-Bench paper v1 · Experimental settings and Tables 3–9 · GameASG-Bench · Strict success, aggregation and cost estimation code
Reasoning settings and token use
The authors vary reasoning effort while keeping DeepSeek-V4-Flash, Claude Code, full tools and the nominal 120-turn setting. Read the task-success and token columns together.
| Effort | Strict success: count/47 (%) | L1 % | L2 P0 % | L2 P1 % | L2 P2 % | Mean reasoning tokens |
|---|---|---|---|---|---|---|
| Low | 7/47 (14.9) | 98.3 | 93.1 | 71.7 | 80.2 | 37,699 |
| High | 19/47 (40.4) | 99.4 | 97.1 | 82.7 | 88.9 | 138,232 |
| Max | 18/47 (38.3) | 99.4 | 98.0 | 86.7 | 90.2 | 189,210 |
Low, High and Max are the reported settings. Reasoning tokens are per-task means. The observed difference between High and Max comes from single runs; it is not evidence of a stable superiority across repetitions.
GameASG-Bench paper v1 · Experimental settings and Tables 3–9
Equal totals can hide different failures
With DeepSeek-V4-Flash, maximum effort and full tools, the two harnesses produce the same success count. Only 10 tasks succeed in both: each harness uniquely succeeds on 8, and neither succeeds on 21.
| Harness | Strict success: count/47 (%) | L1 % | L2 P0 % | L2 P1 % | L2 P2 % |
|---|---|---|---|---|---|
| Claude Code | 18/47 (38.3) | 99.4 | 98.0 | 86.7 | 90.2 |
| Codex CLI | 18/47 (38.3) | 99.2 | 96.1 | 87.1 | 92.2 |
This compares complete harness configurations on the same 47 tasks. Equal strict-success totals do not imply identical artifacts or failure patterns.
GameASG-Bench paper v1 · Experimental settings and Tables 3–9
What the failure traces reveal
Table 9 describes two interaction defects. GPT-6-Astra/Codex CLI’s Diner Dasher passes L1 and P0/P2 but real mouse/touch drags fail to advance service. DeepSeek-V4-Flash/Claude Code’s Turbo Smash Beast stops natural-time simulation after reset or scenario loading, breaking acceleration and coasting checks.
GameASG-Bench paper v1 · Experimental settings and Tables 3–9
What passing can establish
Passing supports compliance with the published requirements under these checks. It does not certify enjoyment, visual quality or production readiness. The test code is public; standard generation containers omit it. NOT_APPLICABLE exclusions and model–harness differences should remain visible when interpreting a comparison.
GameASG-Bench · Task format and execution guide (pinned release) · GameASG-Bench · Applicable-check and priority aggregation code · GameASG-Bench · Armor Alley public gameplay contract (pinned release)
Record the full experiment setup
Pin the repository revision, model, harness, reasoning setting, tool permissions and budget. The runtime pins Claude Code 2.1.206 and Codex CLI 0.153.4. Defaults allow one generation attempt, 3,600 seconds and 120 Claude Code turns. Use a fresh experiment ID; an existing HTML can be re-evaluated. README examples use different defaults from some paper rows and are not exact reproductions.
GameASG-Bench · Pinned runtime image · GameASG-Bench · Generation budgets and container setup · GameASG-Bench · Task format and execution guide (pinned release)