Original September 2026 paper and the authors’ construction environment. This is different from τ-bench and τ²-bench.
What the developer receives
Business records, a simulated client, inherited code and a production API form the construction kit. Requirements must be recovered from evidence and conversation rather than handed over as a complete specification.
Original paper: methods and Table 2 · Authors’ executable environment and task definitions
How submissions are evaluated
The deliverable is an executable customer-service agent. Hidden simulated customer conversations evaluate its behavior after construction. Build time, build cost and serving cost are separate measurements.
Task coverage
The release has 53 construction tasks: six airline, six retail, six telecom and 35 banking tasks. Overall scores average tasks, so banking has the greatest weight.
Published results — original paper Table 2
Overall pass rates: GPT-5.6-sol / Codex 22.0%; GPT-5.6-terra / Codex 18.0%; Claude Opus 5 / Claude Code 23.9%; Claude Sonnet 5 / Claude Code 14.9%; Kimi K3 / Kimi Code 16.1%; Kimi K3 / OpenCode 17.9%. Expert-authored reference agents reach 82.2%. These are paper results, not independent reruns.
How to interpret the comparison
A score belongs to a developer model plus its harness and budget. The expert reference is a ceiling, not another developer model. Simulated clients and domain imbalance limit generalization to live deployments.
Inspect or reproduce the task
Use the authors’ repository for construction kits and the evaluation runtime. Construction runs require Docker and configured model access. Inspect a task and its serving constraints before comparing a new run with the paper.
Published construction and deployment results
Each row describes a developer model and its harness. The expert reference is a separately authored agent, not another developer model.
| Harness | Developer / reference | Overall % | Airline % (6) | Retail % (6) | Telecom % (6) | Banking % (35) | Build minutes | Build USD | Serving / budget |
|---|---|---|---|---|---|---|---|---|---|
| Codex | GPT-5.6-sol (xhigh) | 22.0 | 49.8 | 59.2 | 32.9 | 9.0 | 47.9 | 18.2 | 0.45 |
| Codex | GPT-5.6-terra (xhigh) | 18.0 | 44.5 | 46.6 | 27.2 | 6.9 | 30.0 | 7.0 | 0.38 |
| Claude Code | Claude Opus 5 (max) | 23.9 | 55.9 | 72.8 | 48.2 | 5.9 | 216.3 | 42.0 | 0.58 |
| Claude Code | Claude Sonnet 5 (max) | 14.9 | 41.9 | 50.2 | 21.1 | 3.2 | 235.5 | 30.0 | 0.76 |
| Kimi Code | Kimi K3 (max) | 16.1 | 42.3 | 49.1 | 24.9 | 4.4 | 205.7 | 13.5 | 0.72 |
| OpenCode | Kimi K3 (max) | 17.9 | 43.8 | 52.1 | 27.6 | 6.0 | 360.3 | 15.1 | 0.61 |
| — | Expert-authored reference | 82.2 | 82.3 | 84.0 | 93.8 | 79.8 | — | — | 0.96 |
Overall averages the 53 tasks equally, rather than the four domains equally. The expert reference uses ground-truth access. Serving spend is a credit-budget ratio, not USD; missing reference build values stay unreported.