Game benchmarks: from playing and teamwork to building games

Choose the game task you want an agent to do

Start with the work: controlling a live game, remembering an explored world, reasoning about opponents, coordinating a team, or delivering a working game. This collection links 15 selected benchmark records to their original publications. Use the task groups below to find the evaluation setup that matches your goal.

Live control: act before the scene changes

LOLPlay evaluates screenshot-and-mouse/keyboard agents on 36 champion-specific objectives in native League of Legends. The game continues during inference. Read complete-task success separately from subgoal progress, and check reasoning effort and time budget. The detail page contains the collected results and their conditions.

Orak spans 12 popular games and uses an MCP interface to study agent modules and input modalities. Its single-player game scores and two-player arena results are different views. Inspect each game’s scoring definition before using an aggregate ranking.

OmniGameArena uses twelve Unreal Engine 5 games across Solo, PvP and Coop. It also tracks improvement through reflection rounds and transfer to held-out variants. Separate cold-start performance from performance after feedback and adaptation.

• LOLPlayModel reports
• OrakOpenCompass Hub
• OmniGameArenaClaire Radar

LOLPlay: Benchmarking Real-Time Visual Agents in League of Legends · Orak: A Foundational Benchmark for Training and Evaluating LLM Agents on Diverse Video Games · Orak · official game leaderboard and evaluation notes · OmniGameArena: A Unified UE5 Benchmark for VLM Game Agents with Improvement Dynamics

Exploration and memory: what remains beyond the current screen?

MineExplorer composes Minecraft tasks into exploration goals with implicit prerequisites, using rule-based milestones. It helps examine whether an agent can coordinate dependencies over a longer trajectory, rather than only finish an isolated action.

MirrorCraft compares matched Vanilla and Mirror worlds with hidden changes to server-side rules. Terrain, objectives, interface and action budget are controlled within each pair. Read paired performance changes and whether the agent received the altered rules; familiarity with normal Minecraft mechanics is not enough.

RNG-Bench uses Matching Pairs and a 3D Maze to test reconstruction of earlier observations during interaction. Its Memory Gap analysis separates forgetting from poor action choice. Difficulty and observation modality affect how much history a model must retain.

• MineExplorerClaire Radar
• MirrorCraftClaire Radar
• RNG-BenchClaire Radar

MineExplorer: Evaluating Open-World Exploration of MLLM Agents in Minecraft · MirrorCraft: Paired Evaluation under Hidden Rule Changes in Minecraft · Beyond the Current Observation: Evaluating Multimodal Large Language Models in Controllable Non-Markov Games

Strategy: a good move, a good match, or accurate beliefs?

PokerBench here means the 2025 work by Zhuang and colleagues, with 11,000 pre-flop and post-flop scenarios. Read scenario decision evaluation and the paper’s head-to-head games as separate experiments, with their respective evaluation protocols.

PTCG-Bench uses the Pokémon Trading Card Game to examine decision-making and improvement through accumulated experience. Its modular harness ablations help distinguish the model from the surrounding agent design. Match the harness and experience budget when comparing results.

MafiaScope probes agents’ private assessments during Mafia without inserting those probes into the continuing game context. Role predictions can be checked against engine truth and belief trajectories inspected. Winning, accurate role inference and calibrated confidence are separate observations.

• PokerBenchOpenCompass Hub
• PTCG-BenchClaire Radar
• MafiaScopeClaire Radar

PokerBench: Training Large Language Models to become Professional Poker Players · PTCG-Bench: Can LLM Agents Master Pokémon Trading Card Game? · MafiaScope: Inspecting Multi-Agent Reasoning in Social Deduction Games

Collaboration: succeed with different information and a shared deadline

GPTNT runs in Keep Talking and Nobody Explodes. One agent sees the game puzzle; another has its manual. They must communicate asynchronously against a live countdown. The benchmark evaluates state tracking, communication and timely coordination across the game’s puzzles.

TickingCollabBench creates Minecraft tasks requiring complementary agents under dynamic, time-sensitive conditions. Its generation pipeline filters infeasible configurations. Check the roles, each agent’s observations and the actual timing rules before interpreting a team result.

• GPTNTClaire Radar
• TickingCollabBenchClaire Radar

GPTNT: Benchmarking Real-Time Collaboration Between Multimodal Agents on Keep Talking And Nobody Explodes · Multi-agent Framework for Time-Sensitive Complementary Collaboration in Minecraft

GameASG-Bench: deliver a browser game that passes gameplay checks

GameASG-Bench contains 47 browser-game generation tasks across 12 genres. The deliverable is one executable index.html, built against explicit gameplay requirements and a public testing interface. L1 checks the artifact and source requirements; L2 exercises behavior in a real browser.

A generated file, a completed evaluation run and a fully successful task are distinct outcomes. Read mean check pass rates alongside strict task success: a game can pass many checks while missing a required mechanic. The existing Radar guide preserves the paper’s tables, harness settings and pinned implementation references.

• GameASG-BenchClaire Radar

GameASG-Bench: Benchmarking Autonomous Software Generation for Game Development · GameASG-Bench · task contract and evaluation guide (pinned release)

Engine projects: build, modify, repair and port

GameCraft-Bench contains 140 Godot tasks across 15 game families. It evaluates end-to-end playable artifacts using replayed demonstrations and rubric-guided multimodal assessment. Inspect mechanics, completeness and presentation as well as whether the project launches.

GameEngineBench evaluates 110 scoped C++ implementation tasks from nine Unreal Engine 5 repositories. Changes must compile and satisfy behavioral tests inside executable projects. This is modification of existing runtime systems, with requirements spanning gameplay, UI, networking and other engine behaviors.

SWE-Game contains 247 tasks grounded in 41 reference Godot games. Its five task types cover briefs, design documents, skeleton completion, fault repair and Godot-to-Unity porting. Runtime checks and replay evidence assess behavior; separate visual rubrics assess presentation. Read scores by task type and evaluation dimension.

• GameCraft-BenchClaire Radar
• GameEngineBenchClaire Radar
• SWE-GameClaire Radar

GameCraft-Bench: Can Agents Build Playable Games End-to-End in a Real Game Engine? · GameEngineBench: Evaluating Coding Agents on Real C++ Runtime Environments · SWE-Game: Can Coding Agents Build the Games We Want?

Before comparing results or reproducing a run

Record the benchmark revision, game or engine version, task split, seeds, exact model, agent harness, observation/action interface and tool access. For live games, distinguish wall-clock time from decision steps; for repeated play, state what feedback or experience carries across episodes.

Keep task success, partial progress, wins, per-check pass rates, reward and visual-rubric scores distinct. Save trajectories or project artifacts with the evaluator output. Missing collected scores remain unknown, not zero. Follow each linked detail page for the reported measurements and their provenance.

LOLPlay: Benchmarking Real-Time Visual Agents in League of Legends · GameASG-Bench: Benchmarking Autonomous Software Generation for Game Development · SWE-Game: Can Coding Agents Build the Games We Want? · Orak · official game leaderboard and evaluation notes