Benchmark version 3; public code/data release 2.1.0 (2 November 2026). The two main runs began in October; the page review date is not an experiment date.
Rebuild a published biological model
The agent receives a paper, its extracted text, a focus prompt and a visual manifest. It writes MorpheusML XML, runs simulations, inspects outputs and revises the model within four hours. A separate evaluator assesses biological evidence against criteria tailored to that paper.
15 papers, two model cohorts
The main experiment has 225 GPT-6.1 Sol jobs (15 papers × five modes × three repetitions) and 375 GPT-6 Luna jobs (five repetitions). Subjects use max reasoning effort; the scoring model is GPT-6 Luna at high effort, with scorer v8 and criteria v7. The CSV supplies the 600 original run records.
Pinned author README · methods and printed results · All 600 published run records · Run-table column definitions
Tool access changes the experiment
Five setups separate general Python/web access, a simulator CLI, local documentation, a modeling skill and an MCP tool server. The Docs setup has no internet; the Web setup cannot invoke the simulator itself and receives a host-side execution afterward.
| Setup | Capabilities |
|---|---|
| 1 · Web | Python + web; host executes Morpheus afterward / Python + 网页，之后由主机执行 |
| 2 · CLI | Python + web + Morpheus CLI / Python + 网页 + 模拟器命令行 |
| 3 · Docs | Python + Morpheus CLI + staged docs; no internet / 本地文档，断网 |
| 4 · Skill | Python + web + Morpheus CLI + skill / 增加建模技能 |
| 5 · MCP | Python + web + Morpheus CLI + skill + MCP / 增加 MCP 工具 |
These are protocols, not different benchmark datasets.
Biological fidelity, not merely running code
The primary measurement is a fraction from zero to one: passed criterion weight divided by total weight. The v7 rubric gives qualitative and numerical categories equal total weight. The agent's own 0–8 reproduction score and the host's 0–7 executability score are separate diagnostics and are not the preprint evaluation.
The published headline uses four modes
The README prints mean biological score across all 15 papers and the four Morpheus-enabled modes. The Python/web-only baseline is excluded from this aggregate. We preserve these printed values instead of inventing an all-five-mode headline or converting them to percentages.
| Model | Mean biological score (0–1) | Mean API-equivalent cost per run |
|---|---|---|
| gpt-6.1-sol | 0.850 | US$3.74 |
| gpt-6-luna | 0.663 | US$0.32 |
Cost is the paper's run-cost estimate, not a current price quote. Repetition counts differ.
Two directly printed mode examples
The README also gives the two mode-specific means below. These are narrower protocol slices and must not be ranked against the pooled four-mode headline as though they were the same experiment. All 600 raw run records and the authors' analysis scripts are linked for further inspection; Radar has not rerun the model jobs.
| Model | Setup | Mean biological score (0–1) |
|---|---|---|
| gpt-6.1-sol | MCP | 0.862 |
| gpt-6-luna | Python + web | 0.108 |
These are the two examples explicitly printed in the README; no omitted mode mean is guessed.
Pinned author README · methods and printed results · All 600 published run records
A bounded scientific-reproduction test
This is a 15-paper simulation benchmark, not evidence that an agent can perform wet-lab research or that every simulated mechanism is biologically correct. Results depend on tool access, repetition count, time budget and an LLM judge. The judge shares a model family with one evaluated subject; manual-score comparison data are provided, but do not remove all judge uncertainty.
Pinned author README · methods and printed results · Original Figure 2 evidence and manual comparison
Code, results and release history
The authors call 2.0.0 (27 October) the first public code release. Version 2.1.0 adds figure/data packages while leaving the subject-facing version-3 setup unchanged. Code and per-run result CSVs are public. Complete transcripts, workspaces and all logs are available on request rather than included in this release; four input papers require separate acquisition.
Author code/data release 2.1.0 · First public code release 2.0.0 · Pinned author README · methods and printed results