Model reports

SWE-bench Science

Repository-level scientific software engineering benchmark from the OpenMOSS team (Fudan): 119 tasks across 98 GitHub repositories in 20 scientific domains, organized into Issue-driven, Expert-exploratory, and Engineering-integration paradigms. Best reported pass@1 is below 50% (Claude Code with Opus-5 max), and the paper identifies four recurring failure mechanisms, so scores are highly sensitive to scientific-domain knowledge and tool access, not only to coding ability.