Paper v1, submitted 30 October 2026; 108 scaffolded ROS1-to-ROS2 migration tasks.
What the model receives
Complete missing ROS2 logic from scaffolded ROS1 files, task instructions and migration context.
108 executable tasks
The suite contains 44 interface, 29 behavior and 35 system tasks; target files cover C/C++, Python and configuration.
Tests stay outside candidate control
Only target files replace reference-package files. Tests and build configuration remain fixed; passing requires a successful build and every runtime test to run successfully.
Single-attempt success
pass@1 uses five generations per task at temperature 0.2 via OpenRouter. Capability categories are separate subsets; pass@3 and pass@5 are different sampling budgets.
| Model | pass@1 % [95% CI] | pass@3 % | pass@5 % | Interface % | Behavior % | System % |
|---|---|---|---|---|---|---|
| Claude Opus 4.6 | 10.9 [5.6,16.9] | 12.5 | 13.0 | 8.2 | 11.7 | 13.7 |
| GPT-5.5 | 10.0 [5.4,15.2] | 14.6 | 16.7 | 9.5 | 12.4 | 8.6 |
| Gemini 3.1 Pro | 7.2 [3.5,11.5] | 11.5 | 13.9 | 5.5 | 7.6 | 9.1 |
| Qwen 3.5 Plus | 6.7 [3.0,11.1] | 9.3 | 10.2 | 6.4 | 3.4 | 9.7 |
| GLM-5 Turbo | 6.5 [2.6,10.9] | 8.2 | 8.4 | 4.5 | 6.2 | 9.2 |
| DeepSeek V4 Pro | 5.7 [2.2, 9.8] | 8.6 | 9.4 | 4.6 | 3.5 | 9.1 |
Printed values; all-task and subset columns have different denominators.
All eleven task categories
Figure 4 provides pass@1 percentages for each model and category. These subset scores explain behavior; only the full-suite pass@1 scores enter the primary results series.
| Category | Claude Opus 4.6 | GPT-5.5 | Gemini 3.1 Pro | Qwen 3.5 Plus | GLM-5 Turbo | DeepSeek V4 Pro |
|---|---|---|---|---|---|---|
| Action server / 动作服务端 | 18.3 | 11.7 | 8.3 | 8.3 | 8.3 | 8.3 |
| Manipulation / 操作 | 10.0 | 8.0 | 6.0 | 10.0 | 8.0 | 10.0 |
| Multi-node coordination / 多节点协调 | 20.0 | 13.3 | 17.8 | 15.6 | 15.6 | 15.6 |
| Navigation / 导航 | 12.5 | 0.0 | 0.0 | 0.0 | 0.0 | 7.5 |
| Parameter server / 参数服务端 | 6.2 | 12.3 | 7.7 | 6.2 | 4.6 | 4.6 |
| Perception and control / 感知与控制 | 15.6 | 11.1 | 6.7 | 8.9 | 11.1 | 4.4 |
| Publisher/subscriber / 发布／订阅 | 6.7 | 13.3 | 4.4 | 11.1 | 4.4 | 4.4 |
| Sensor processing / 传感器处理 | 18.2 | 18.2 | 14.5 | 1.8 | 7.3 | 5.5 |
| Service client / 服务客户端 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 |
| Simulation integration / 仿真集成 | 12.5 | 12.5 | 12.5 | 12.5 | 12.5 | 2.5 |
| State machine / 状态机 | 0.0 | 6.7 | 0.0 | 0.0 | 0.0 | 0.0 |
All 66 printed values were transcribed from the original figure; zeros are reported values.
Repair is a separate experiment
Execution-feedback repair reports cumulative successes, not the zero-shot pass@1 metric. Fractions below remain unconverted.
| Model | Initial | Round 1 | Round 2 | Round 3 / final |
|---|---|---|---|---|
| Claude Opus 4.6 | 12/108 (11.1%) | 32/108 (29.6%) | 44/108 (40.7%) | 53/108 (49.1%) |
| GPT-5.5 | 12/108 (11.1%) | 31/108 (28.7%) | 41/108 (38.0%) | 51/108 (47.2%) |
| Qwen 3.5 Plus | 7/108 | 23/108 | Not reported / 未报告 | 28/108 (25.9%) |
The Qwen intermediate round is not supplied in the text.
Scope and unresolved details
This tests middleware, not robot motion or control quality. Model-generated references need adequacy checks. Three generation slots were absent. Exact provider revisions and run dates remain unspecified.
Source and release status
The paper is accessible. Its advertised repository returned HTTP 404 during this review, so code availability is unverified.