September 2026 paper v1 and the authors’ public task site; the main experiment uses GUI interaction.
What work does it cover?
There are 200 tasks across 11 workflows, including sketches, parts, assemblies, CAM, FEM and technical drawings. The suite is largest in part modelling and sketching; small categories should be read with their task counts.
How the agent works
The main setup permits screenshots and GUI actions, with 100 GUI steps per task. A separate terminal experiment uses 50 tasks and should not be pooled with these results.
Authors’ project · tasks, evaluation and reproduction links · CADWorld introducing paper · protocol, evaluation and limitations (arXiv v1)
The saved artifact decides success
Executable checks inspect native .FCStd projects and required auxiliary files. Geometry, constraints, feature structure or downstream outputs must satisfy every task requirement; diagnostics do not award partial credit.
CADWorld introducing paper · protocol, evaluation and limitations (arXiv v1)
What the published comparison shows
The best reported agent succeeds on 17.5% of tasks, versus an 87.0% expert reference. The expert row represents one reference pass, not a study of average engineer performance.
What remains outside the test
Tasks provide explicit goals and assets. FreeCAD-only results do not establish performance in other CAD products, open-ended client projects, or requirements discovery.
CADWorld introducing paper · protocol, evaluation and limitations (arXiv v1)
Start from a task and its checks
The official site links the tasks, VM and installation guide. Reproduction needs Linux, KVM, Docker and the FreeCAD VM; preserve the harness, budget and trace alongside each reported result.
Published task-success results
Success requires all task checks to pass. These are the success values from Table 3, distinct from its finish, cost and action-count columns.
| Agent / reference | Task success |
|---|---|
| GPT5.4 (CUA) | 17.5% |
| Opus4.8 (CUA) | 16.0% |
| Kimi K2.6 | 7.5% |
| OpenCUA | 1.5% |
| Holo 3.1 | 0.5% |
| Qwen3.6 | 0.0% |
| MiniMax M3 | 0.0% |
| Human Expert reference | 87.0% |
Names and percentages follow the paper. The human row is a single expert reference pass; it is not an average across engineers. The terminal-only 50-task ablation is excluded.
CADWorld introducing paper · protocol, evaluation and limitations (arXiv v1)