The 47-task v1 suite and fixed 10-task v1-lite subset. September 2026 corrected Medal Scores are separate from the original paper’s average ranks.
An engineering optimization loop
The agent starts from working code, proposes edits and receives objective values and constraint feedback from task-specific simulators or verifiers. The 47 tasks span five categories, including computing, decision science, control, communications and physical engineering. Examples include battery charging and memory allocation.
Frontier-Engineering official project and correction notice · Frontier-Eng paper v2 · evaluation protocol and Table 2
What the Medal Score means
Per task, meeting the available gold, silver or bronze threshold earns 1.00, 0.67 or 0.33. The mean over 47 or 10 tasks stays on a 0–1 scale. Invalid results earn zero; absent medal tiers earn no credit. This is relative to published peer thresholds, not the percentage of engineering problems solved.
Official leaderboard · corrected scores and historical ranks · Pinned released medal scoring documentation
The September corrections matter
The authors strengthened evaluator isolation and corrected validation problems. The September 15 scores combine re-evaluated submissions with replacement candidates from shorter reruns; they are not a fresh equal-budget experiment. Keep v1 and v1-lite separate. The original nine-model rank table also includes gpt-oss-120b, whose raw scores were not retained for the updated eight-model podium.
Frontier-Engineering official project and correction notice · Official leaderboard · corrected scores and historical ranks
Published corrected scores
All eight published models are shown below. The two columns use different task sets.
| Model | v1 · 47 tasks | v1-lite · 10 tasks |
|---|---|---|
| Claude Opus 4.6 | 0.533 | 0.501 |
| GPT-5.4 | 0.454 | 0.267 |
| GLM-5 | 0.347 | 0.300 |
| Gemini 3.1 Pro Preview | 0.284 | 0.300 |
| DeepSeek V3.2 | 0.269 | 0.299 |
| Grok 4.20 | 0.220 | 0.167 |
| Seed 2.0 Pro | 0.206 | 0.100 |
| Qwen3 Coder Next | 0.170 | 0.066 |
Higher is better; 0–1. Preserve the corrected-evaluator protocol above.
Official leaderboard · corrected scores and historical ranks
Historical average rank
The original paper uses within-task ranks averaged across 47 tasks, under OpenEvolve with 100 iterations; GPT-5.4 uses a full-suite retest. This lower-is-better metric is neither the current Medal Score nor a new measurement after the September fixes.
| Model | Mean rank |
|---|---|
| GPT-5.4 | 3.54 |
| Claude Opus 4.6 | 3.63 |
| GLM-5 | 4.34 |
| DeepSeek V3.2 | 4.76 |
| gpt-oss-120b | 4.81 |
| Gemini 3.1 Pro Preview | 5.53 |
| Grok 4.20 | 5.82 |
| Seed 2.0 Pro | 5.86 |
| Qwen3 Coder Next | 6.71 |
Original evaluators; 9 models; lower is better. Retained only as an attributed historical table.
Frontier-Eng paper v2 · evaluation protocol and Table 2 · Official leaderboard · corrected scores and historical ranks
Read a score with the environment
Task objectives have different physical units, so raw per-task scores should not be averaged directly. Hardware, simulator dependencies, search method and evaluator version matter. The original paper’s historical curves do not establish that invalid-solution problems were absent; the later corrections are part of the record.
Frontier-Eng paper v2 · evaluation protocol and Table 2 · Official repository · environments and version branches · Frontier-Engineering official project and correction notice