Version 1.0; dated author-analysis task counts and a separately labelled live leaderboard captured 3 November 2026.
Can an agent finish a cloud change without breaking its neighbors?
CloudOpsBench asks an agent to change an existing AWS-style environment: retire a service, restore a broken path, move a workload or restrict access. The instruction names the desired outcome; the agent must discover the relevant resources. Passing requires the requested final state and preservation of resources outside the task.
CloudOpsBench cloud-operations task explanation · 13 October 2026 · CloudOpsBench official task repository
45 tasks across five operational workflows
The authors describe version 1.0 as 45 tasks. These are executable changes to cloud state, with setup, instructions, automated checks and reference solutions in the public task repository. They do not measure a model’s ability to answer a cloud certification quiz.
| Workflow | Tasks |
|---|---|
| Decommissioning / 资源下线 | 13 |
| Security & access / 安全与权限 | 12 |
| Migration & deployment / 迁移与部署 | 7 |
| Configuration & monitoring / 配置与监控 | 6 |
| Repair & recovery / 修复与恢复 | 7 |
Counts describe the benchmark inventory, not model successes.
CloudOpsBench author failure analysis · 2 November 2026 · CloudOpsBench official task repository
A fresh environment and a check of the final state
The current method page specifies four runs per task, 180 planned trials per model–effort–harness entry and a 40-minute agent shell budget. Agents receive AWS CLI, boto3 and Terraform in a non-root shell. The verifier checks cloud state rather than accepting the final chat message. A failed setup or verifier error is recorded separately; resolution rate divides passes by passes, failures and timeouts, excluding evaluation errors.
CloudOpsBench official evaluation method · CloudOpsBench cloud-operations task explanation · 13 October 2026
What the dated result counts mean
The 2 November 2026 author analysis explicitly states that Claude Fable 5.1 completed 29 of 45 tasks and GPT-6 Astra completed 26. Those two printed counts are the primary observations here. The article does not define how these counts aggregate repeated runs, nor identify the exact effort and harness beside each count. They are not converted to percentages and are not equated with the live leaderboard’s trial-level resolution rate. November 2 is the analysis publication date, not an inferred experiment date.
| Model label in article | Completed tasks | Task inventory |
|---|---|---|
| Claude Fable 5.1 | 29 | 45 |
| GPT-6 Astra | 26 | 45 |
Effort, harness and aggregation across runs are not specified alongside these counts.
Live leaderboard: keep effort and harness visible
The following 18 rows reproduce the displayed leaderboard values observed on 3 November 2026. Model names, reasoning effort, harness, time, tokens and API cost remain visible. The page’s Release date column is a model-release date, not an experiment or result-publication date; this source does not expose an exact run date for each aggregate. These undated percentages remain in this cited table rather than receiving invented dates in the primary score history.
| Model (effort) | Harness | Resolution rate | Time | Tokens | API cost (USD) |
|---|---|---|---|---|---|
| claude-fable-5-1 (max) | claude-code | 60.6% | 38.6h | 97.9M | $404.3 |
| gpt-6-astra (max) | codex | 58.9% | 16.5h | 98.0M | $310.9 |
| gemini-4-argon (high) | gemini-cli | 57.5% | 34.2h | 173.9M | $100.5 |
| grok-4.7 (high) | cursor-cli | 56.5% | 33.2h | 178.3M | $138.8 |
| gpt-6-astra (high) | codex | 54.2% | 16.1h | 62.4M | $188.4 |
| grok-4.7 (high) | grok-build | 48.3% | 25.9h | 173.4M | $127.3 |
| claude-opus-5-5 (high) | terminus-2 | 47.8% | 19.5h | 22.6M | $45.1 |
| claude-opus-5-5 (high) | claude-code | 46.7% | 14.0h | 37.4M | $36.6 |
| claude-fable-5-1 (high) | terminus-2 | 43.8% | 19.0h | 37.3M | $319.1 |
| gemini-3.8-flash (high) | terminus-2 | 42.8% | 33.2h | 337.4M | $70.0 |
| gpt-6-astra (high) | terminus-2 | 42.8% | 12.9h | 19.2M | $94.6 |
| grok-4.7 (medium) | grok-build | 40.0% | 21.4h | 121.6M | $91.6 |
| grok-4.6 (high) | cursor-cli | 39.7% | 19.7h | 78.2M | $63.8 |
| grok-4.6 (medium) | grok-build | 36.7% | 16.2h | 59.4M | $51.2 |
| grok-4.6 (high) | grok-build | 36.1% | 17.7h | 72.0M | $61.7 |
| qwen3.8-max (high) | terminus-2 | 33.7% | 79.1h | 78.4M | $61.8 |
| deepseek-v4.1-flash (high) | terminus-2 | 21.8% | 22.2h | 68.1M | $7.5 |
| kimi-k3.1-pro (high) | terminus-2 | 19.1% | 14.8h | 7.4M | $10.7 |
Values and rounding are preserved as displayed. Time, tokens and cost are leaderboard totals for the entry, not a per-task price. Resolution rate excludes evaluation errors. Capture date does not establish run date; these percentages use a different metric from the two task counts above.
CloudOpsBench official v1.0 leaderboard · CloudOpsBench official evaluation method
The authors’ failure analysis is a separate view
The dated analysis attributes approximately 80% of failures to reasoning or scope mistakes and approximately 20% to wrong actions, and reports no knowledge-gap failures in its inspected set. These are the authors’ qualitative labels and approximate proportions, not an independent audit or a general finding about every cloud task. A system can issue individually successful commands and still miss a resource or damage unrelated state.
What is available to inspect or reproduce
The public repository provides task definitions, setup scripts, verifiers and reference solutions. Its README says users supply the execution harness; validating task structure does not rerun the leaderboard. The method page describes either an emulator or a real cloud environment and publishes redacted trajectories and grading outputs subject to its disclosure rules. Radar has inspected the published material, not rerun cloud operations or independently established emulator–AWS equivalence.
CloudOpsBench official task repository · CloudOpsBench official evaluation method