August 2026 paper v1; parametric-part and assembly tracks retain separate metrics and denominators.
Two tracks, different deliverables
P uses 300 parts for 600 generation/editing tasks. A contains 150 body pairs; its reported comparison excludes 30 development pairs and evaluates 120.
CADEngBench introducing paper · tracks, metrics and limitations (arXiv v1)
From valid solids to controlled edits
L0 checks execution and STEP round-tripping; L1 checks engineering requirements; L2-Z tests parameter behaviour; L2-E tests the requested edit while preserving protected geometry.
CADEngBench introducing paper · tracks, metrics and limitations (arXiv v1)
A solved simulation is only the start
L3 covers 164 eligible parts. CalculiX compares generated and reference responses under matched conditions; simulation reach and successful matched-state pairs use different denominators.
CADEngBench introducing paper · tracks, metrics and limitations (arXiv v1)
Identify the joint, entities and motion
Assembly scoring distinguishes entity retrieval from joint-type correctness. End-to-end success also requires a correct frame and verified permitted and blocked motions.
CADEngBench introducing paper · tracks, metrics and limitations (arXiv v1)
Read the scope before comparing scores
Tests cover bounded edits, body pairs and linear-static analysis. They do not certify manufacturing readiness or full-assembly recovery.
CADEngBench introducing paper · tracks, metrics and limitations (arXiv v1)
What can be reproduced from the public package?
The authors’ repository provides ten P development items and seven A development pairs, not the sealed full corpora. It includes evaluator code, sample controls and saved reference summaries. Full-paper aggregates require the frozen full datasets and original model responses.
Reproduction prerequisites
The package specifies Python 3.11, CalculiX 2.21 and an Ubuntu/Linux reference environment. Its verification and control scripts check the installation before model calls; Apple Silicon needs an additional conda-compatible environment for PyBullet.
Published results: parametric parts
Part-generation, editing and simulation columns describe different checks. Parentheses retain the paper’s evaluable counts.
| System | L0 % | L1 % | L2-Z % (evaluable) | L2-E % | L3 reach % (parts/164) | L3 pair pass % (passed/pairs) |
|---|---|---|---|---|---|---|
| GPT-5.2 | 41.3 | 22.0 | 31.6 (291) | 70.3 | 37.8 (62/164) | 32.8 (65/198) |
| Claude 4.5 | 58.0 | 28.3 | 41.4 (290) | 66.7 | 54.3 (89/164) | 35.8 (101/282) |
| Gemini 3 Flash | 52.7 | 30.0 | 39.8 (289) | 72.3 | 46.3 (76/164) | 46.3 (119/257) |
| GLM-4.6V | 47.3 | 13.3 | 30.7 (287) | 59.3 | 44.5 (73/164) | 27.0 (64/237) |
| Kimi K2.5 | 53.3 | 24.3 | 41.0 (288) | 67.0 | 51.8 (85/164) | 34.8 (94/270) |
| Mistral 3.5 | 29.3 | 8.3 | 16.5 (297) | 68.3 | 26.8 (44/164) | 34.5 (49/142) |
| Llama 4 Maverick | 48.0 | 12.7 | 34.0 (288) | 60.7 | 47.6 (78/164) | 27.5 (71/258) |
| Qwen3.5-35B | 13.3 | 5.0 | 6.3 (300) | 61.7 | 15.9 (26/164) | 36.6 (30/82) |
L0, L1 and L2-E each use 300 items; L2-Z uses the shown per-model denominator. L3 reach counts parts; L3 pair pass counts comparable generated/reference parameter states. Do not average these columns into a single score.
CADEngBench introducing paper · tracks, metrics and limitations (arXiv v1)
Published results: assembly relations
Entity retrieval, typed retrieval and end-to-end motion checks remain separate. MRR uses a different scale from percentages.
| System | Valid % | Entity@1 % | Typed@1 % | Typed@3 % | MRR | A2 E2E % (passed/120) |
|---|---|---|---|---|---|---|
| GPT-5.2 | 100.0 | 61.7 | 12.5 | 30.0 | .197 | 11.7 (14) |
| Claude 4.5 | 100.0 | 57.5 | 14.2 | 32.5 | .221 | 12.5 (15) |
| Gemini 3 Flash | 100.0 | 62.5 | 23.3 | 41.7 | .310 | 15.8 (19) |
| GLM-4.6V | 99.2 | 50.0 | 10.0 | 27.5 | .174 | 8.3 (10) |
| Kimi K2.5 | 78.3 | 56.7 | 15.0 | 32.5 | .228 | 7.5 (9) |
| Mistral 3.5 | 100.0 | 52.5 | 15.8 | 25.8 | .200 | 14.2 (17) |
| Llama 4 Maverick | 100.0 | 55.8 | 14.2 | 26.7 | .192 | 10.8 (13) |
| Qwen3.5-35B | 100.0 | 55.0 | 14.2 | 24.2 | .190 | 10.8 (13) |
The remaining 30 of the 150 source pairs were reserved for evaluator development. Entity ignores joint type; Typed also requires joint family and body roles. The final column retains the paper’s percentage and parenthesized pass count.
CADEngBench introducing paper · tracks, metrics and limitations (arXiv v1)