Original arXiv v1, with 36 tasks and nine model–effort configurations; the submission history displays 25 October 2026. Experiment dates are not specified.
Control a live game from screenshots
LOLPlay gives the agent one champion-specific objective at a time in a native League of Legends custom match. It reads screenshots and recent history, then sends mouse and keyboard actions. The game keeps running during requests and action execution; setup and scoring APIs are not supplied as the agent’s observation.
36 tasks and 82 checked requirements
The suite spans six primary task families, one objective per champion. Requirements can overlap. Twenty-six tasks have no bot champions; ten include beginner enemy bots and three of those also include an allied bot. This evaluates bounded objectives rather than ranked matches or full-match win rate.
| Task family | Tasks | Subgoals |
|---|---|---|
| Economy and farming | 6 | 7 |
| Development | 9 | 26 |
| Sustain | 7 | 18 |
| Combat | 6 | 13 |
| Cooperation | 3 | 8 |
| Map objectives | 5 | 10 |
| Total | 36 | 82 |
Families describe primary demands; the original task catalog gives every champion objective.
One run per task and model–effort setting
Three models at low, medium and high reasoning effort each run all 36 tasks: 324 runs in total. Task budgets range from 5 to 20 minutes, capped at 100 model responses. One response may contain multiple sequential actions. History keeps the latest three valid turns without summarization; initialization failures stay in the denominator. Exact experiment dates, provider snapshot identifiers and the game patch are not stated.
Completion and partial progress are different measurements
The primary score is confirmed task passes divided by all 36 tasks, expressed as a percentage. All requirements, timing and task context must be valid. The separate subgoal rate first computes the fraction passed within each task, then averages equally across tasks. It is not the pooled percentage over 82 checks, and a 100% subgoal cell alone does not guarantee full task success.
All nine published configurations
Astra low and medium both complete nine tasks (25.0%); medium improves partial progress without adding a completed task. No model improves full-task success at high effort over low effort in this experiment. These are descriptive comparisons under the reported interface and budgets; response latency includes service, network, retries and parsing, not reasoning time alone.
| Model | Effort | Task success (%) | Task-averaged subgoal success (%) | Mean responses | Mean latency (s) |
|---|---|---|---|---|---|
| GPT-6 Astra | Low | 25.0 | 37.7 | 43.9 | 7.47 |
| GPT-6 Astra | Medium | 25.0 | 40.3 | 37.8 | 8.39 |
| GPT-6 Astra | High | 16.7 | 33.7 | 27.1 | 16.67 |
| GPT-6.1 Sol | Low | 13.9 | 32.1 | 42.8 | 8.90 |
| GPT-6.1 Sol | Medium | 11.1 | 31.3 | 33.2 | 11.44 |
| GPT-6.1 Sol | High | 11.1 | 24.0 | 37.4 | 10.26 |
| DeepSeek-v4.1-Flash | Low | 5.6 | 22.2 | 89.8 | 2.81 |
| DeepSeek-v4.1-Flash | Medium | 2.8 | 20.0 | 91.9 | 3.05 |
| DeepSeek-v4.1-Flash | High | 0.0 | 18.1 | 77.0 | 4.67 |
All primary percentages use the full 36-task denominator. The zero for DeepSeek high is an observed result, not missing data.
What the partial successes contain
Across all configurations, 144/567 positive-attainment checks pass (25.4%), compared with 53/117 avoidance checks (45.3%) and 46/54 state-preservation checks (85.2%). The pooled total is 243/738 (32.9%); the task-averaged rate is 28.8%. Avoiding death or leaving an ability unlearned may satisfy a requirement without advancing the rest of the objective.
| Model | Effort | Positive attainment: passed/eligible (%) | Avoidance: passed/eligible (%) | State preservation: passed/eligible (%) |
|---|---|---|---|---|
| GPT-6 Astra | Low | 26/63 (41.3) | 4/13 (30.8) | 5/6 (83.3) |
| GPT-6 Astra | Medium | 25/63 (39.7) | 5/13 (38.5) | 5/6 (83.3) |
| GPT-6 Astra | High | 21/63 (33.3) | 5/13 (38.5) | 5/6 (83.3) |
| GPT-6.1 Sol | Low | 21/63 (33.3) | 4/13 (30.8) | 5/6 (83.3) |
| GPT-6.1 Sol | Medium | 18/63 (28.6) | 4/13 (30.8) | 6/6 (100.0) |
| GPT-6.1 Sol | High | 17/63 (27.0) | 4/13 (30.8) | 4/6 (66.7) |
| DeepSeek-v4.1-Flash | Low | 7/63 (11.1) | 9/13 (69.2) | 4/6 (66.7) |
| DeepSeek-v4.1-Flash | Medium | 5/63 (7.9) | 9/13 (69.2) | 6/6 (100.0) |
| DeepSeek-v4.1-Flash | High | 4/63 (6.3) | 9/13 (69.2) | 6/6 (100.0) |
These are pooled requirement counts, not the equally weighted task scores in Table 6.
What happens at the response limit
This slice includes only DeepSeek-v4.1-Flash runs stopped at 100 responses. Low and medium effort reach the cap after a median 64.0% and 63.7% of their time allowance. Frequent responses can coexist with little progress; these conditional counts cannot establish what would happen with the cap removed.
| Effort | Capped runs | Successful tasks | Zero-subgoal runs | Positive checks passed/eligible | Avoidance/preservation passed/eligible |
|---|---|---|---|---|---|
| Low | 26 | 1 | 13 | 4/46 | 12/16 |
| Medium | 29 | 0 | 19 | 1/47 | 11/14 |
| High | 14 | 0 | 8 | 0/24 | 8/9 |
Task and zero-subgoal columns count runs; the final columns count checks. This is a subset of the main evaluation.
Stopping is not the evaluator’s verdict
The 324-run cohort has 40 successful and 284 unsuccessful runs. A model’s DONE message, its response limit or its time limit does not itself determine success. The table lists the separate runner reasons. Completed-response latency across 17,221 responses has mean 6.536 s, median 5.594 s and 95th percentile 14.031 s; requests with no completed response are excluded.
| Reason | Count |
|---|---|
| done | 37 |
| infeasible | 60 |
| execution_budget_exhausted | 129 |
| max_steps | 75 |
| execution_error | 22 |
| setup_error | 1 |
| no_actions | 0 |
| environment_terminated | 0 |
These eight rows partition the same 324 runs. Their counts must not be added to the separate 40/284 success split.
Small, task-specific evidence
The cohort has no successful sustain, cooperation or map-objective task, and no passed personal-kill or allied-assist check. Two experienced players validated feasibility of all 36 tasks; their 100% completion is a feasibility check, not a population human baseline. One task per champion and a single run per configuration limit generalization. The paper’s task matrix is linked for the full 36-by-nine diagnostic breakdown.
Original paper v1 · methods and Tables 2, 6, 9, 11, 12 · Original Figure 8 · full task diagnostics
Original code and publication
The public author repository includes task definitions, setup, desktop control and independent scoring. Its gameplay runner targets Windows with a signed-in League client, required champions/bots and model credentials. The repository offers five agent modes; the nine paper results are model–effort configurations under the paper’s common screenshot-to-action protocol, not evidence for every code mode. The arXiv submission history explicitly dates v1 to 25 October 2026 despite its 2611 identifier. Exact experiment dates remain unknown; this page was reviewed on 4 November.
Author repository README at reviewed commit · arXiv original submission history